Language selection

Government of Canada / Gouvernement du Canada

Search

CSPS Strategic Directions Initiative Integrated Summative Evaluation: Summary Report

May 2018

Integrated Summative Evaluation – Final – Summary Report

 Report summary

Background and context

In November 2014, the Treasury Board (TB) approved the Strategic Directions Initiative (SDI, or the Initiative) as a phased project over 3 years (2014–15 through 2016–17) to examine the delivery and management of learning at the Canada School of Public Service (CSPS, or the School), as well as programming, the underlying business model, and the supporting strategies needed to implement a new model. The Initiative aimed to put in place a more accessible, relevant and responsive learning platform. The Initiative included plans to modernize the School's business operations and change the funding model to establish a new approach to learning. In this new approach, departments would focus on mandate-specific learning while the School would deliver a relevant, common curriculum for the public service, and advances a public service-wide culture of learning.

The past 3 years saw considerable activity in terms of consultations with federal government departments and agencies, the phasing in of the new funding model, the revision of the curriculum, and the development of supportive learning technologies and business intelligence systems. As of the end of 2016–17, public servants can now access the School's online platform, GCcampus, with its suite of learning products, from anywhere internet access is provided.

Evaluation purpose and scope

An evaluation of the Initiative was undertaken to provide neutral, credible, and timely feedback to School decision-makers based on the commitments made to the Prime Minister and the Treasury Board in 2014. Two evaluation exercises were conducted. A formative evaluation providing a mid-term assessment of the Initiative was delivered in December 2016. The formative evaluation focused on implementation and addressed questions primarily related to SDI activities, outputs and immediate outcomes. The CSPS Strategic Directions Initiative: Integrated Summative Evaluation focuses on questions related primarily to intermediate and long-term outcomes; specifically, the summative evaluation examined, post-SDI, the adequacy of School funding and the appropriateness of School authorities, the extent to which learner satisfaction, knowledge and application of knowledge increased, the relevance of the School's curriculum, the impact of SDI on the range of CSPS learning products, the extent to which a common, core curriculum was established, and the School's performance in terms of reporting on training outcomes. The summative evaluation also examined the question of how the School can measure, in the future, the outcomes of its learning ecosystem.

Lines of evidence

Findings from 5 lines of evidence were synthesized to address each outcomes review and funding model review question.

Data collection methods included a document review (over 150 documents were reviewed, including internal School documents ranging from project plans to reports, as well as various tracking tools) and the collection of related administrative data pertaining to product registrations and evaluations recorded by the School from 2013–14 through 2016–17. Interviews were conducted with 45 key informants (17 external senior interviewees, 18 additional external interviewees and 10 School interviewees). Finally, a literature review was conducted on methods to measure and report on the learning outcomes of the School's learning ecosystem. A review of the School's funding model was also conducted as a 5th line of evidence.

Key evaluation findings

Key evaluation findings. The table contains two columns. Read down the first column for the evaluation questions, then read to the right for the key findings.
Evaluation question Key findings

Was the amount reallocated to the School aligned with the level of ongoing expenditures required to deliver the new business model (EQ2)?

The School's reference levels for the years following the transition period (2017–18 and beyond) appear to be aligned with projected expenditure requirements.

Are the School's revenue and spending authorities appropriate (EQ3)? Should the School's statutory mandate and revenue and spending authorities be adjusted to reflect its new mandate and business model (EQ1)?

The School's revenue and spending authorities are appropriate and do not need adjustment.

To what extent did Strategic Directions impact the overall range of learning products (EQ8)?

Under SDI the overall range of learning products changed substantially.

Department and agency representatives generally expressed an appreciation of the renewed range of learning products, recognizing that it is an ongoing process and reminding the School of the continued value of face-to-face learning in certain circumstances.

Has a common, core curriculum been established (EQ10)?

The School has successfully established a core common curriculum as identified by the CSPS Advisory Committee.

Department and agency officials generally appreciate the increased clarity that has accompanied the renewed curriculum, but still find limitations in the operationalization of the definition of "core common curriculum."

Is the School's curriculum relevant to public servants and the Government of Canada (EQ7)?

The School's curriculum, as of the end of the 3-year SDI period, was found to be relevant to public servants and the Government of Canada.

Is there an increase in learner satisfaction (EQ5b)?

Learner satisfaction with self-paced online learning products increased over the 3-year SDI period. Satisfaction with other learning modes remained stable.

Is there an increase in the level of knowledge acquired by learners, and are they applying what they learn on the job (EQ12)?

Limited School research revealed learner knowledge gains and positive impacts on learner job effectiveness following some learning activities, but these observations are not generalizable to the range of School products.

Can the School provide evidence of the outcomes of training it delivers (EQ11)?

As a result of the activities undertaken as part of the Initiative, the School has in place an upgraded business intelligence system that, combined with existing data collection practices, is capable of providing a comprehensive range of outcome metrics.

Various iterations of external report models have yet to satisfy the needs of client departments and agencies.

How can the School measure the outcomes of the evolving learning ecosystem (EQ13)?

A system that appropriately and effectively measures the outcomes of the School's evolving learning ecosystem will need to go beyond traditional learning evaluation concepts and data collection approaches.

The School lacks a comprehensive, strategic model by which methodologies are adopted in line with evaluation needs that are themselves determined by the anticipated needs of decision-makers.

Conclusions and recommendations

  1. The new CSPS funding model has been successfully adopted, and planned reference levels and spending authorities appear to be appropriate. Because activities undertaken as part of SDI are still stabilizing and will continue to do so for at least 2 more years, and future demand is difficult to predict so early in the life span of the new model, it is premature to assess with certainty the future resource requirements of the School. That said, it would appear that current reference levels for 2017–18 and beyond will be sufficient in the near term, with the caveat that further augmentations in School responsibilities might need to be accompanied by additional funding.

    Retaining the authority to collect fees and to carry over revenues from these fees to the next fiscal year, provides an appropriate level of flexibility for the School to efficiently and effectively implement the new business model.

    Recommendation 1. It is recommended that, in future, estimated costs associated with the development and delivery of learning become a distinct element of consideration in Treasury Board submissions and Cabinet requests.

  2. The School has established a relevant core common curriculum, as defined by senior officials. The School made substantial progress toward the establishment of a relevant core common curriculum. The School's focus is seen to have been sharpened, and department and agency representatives have greater confidence that the School can meet basic learning needs, leaving to departments and agencies mandate-specific training.

    The core common curriculum is defined, in essence, as the topics and priorities deemed to be relevant to the performance responsibilities of a critical mass of public servants enterprise-wide, along with priorities identified at senior levels. The definition has not been systematically validated at the front-line level, however. Also, the idea of learner needs having to be "enterprise-wide" appears to have precluded training for significant subpopulations; officials in some departments and agencies believe their employees' needs fall largely outside the purview of the School's new definition, and are left having to provide required training themselves.

    Recommendation 2. It is recommended that the School, in consultation with senior officials as well as middle managers and front-line staff across all regions, continues to clarify, validate and refine the definition of its core common curriculum.

  3. The School's range of learning products was extensively overhauled, improving accessibility and choice among learning modes. Under SDI the range of CSPS learning products changed substantially. The number of classroom courses fell from 262 to 87 while the number of events increased nearly fivefold to 372. The number of self-paced online learning products remained relatively stable (at 219 as of the end of 2016–17), but almost all were replaced with newer versions. In addition, new types of learning products were made available.

    Department and agency representatives are generally positive respecting the renewed range of learning products, noting significant improvements in terms of learning mode choice and in the accessibility of learning opportunities. The School appears better able to respond rapidly to emergent priorities. Concerns remain, however, related to the shift from a model emphasizing classroom-based training to one relying more on online learning activities. Fewer classroom options can lead to perceptions among users about possible seat shortages. Also, an overreliance on online modes can impede training for employees without ready access to the internet.

    Recommendation 3. It is recommended that the School continues to develop and optimize the range of learning modes associated with its learning ecosystem.

  4. The School has yet to establish a robust outcomes measurement regime. CSPS has collected Level 1 evaluation data since long before SDI, and found through its Level 1 data tracking efforts that satisfaction with self-paced online learning products increased over the SDI period. Level 1 observations are, however, of the least interest to department and agency representatives who are keener to know the extent to which their employees are acquiring new knowledge and skills from their School experiences and the extent to which they are applying these learnings on the job.

    The School collects elementary knowledge acquisition and learning application data on most activities for which registration is required, plus detailed data on a small sample of learning activities each year, but it has not developed an analysis protocol sufficient to satisfy client organizations. Moreover, in light of the evolution of the School's learning ecosystem, a system that appropriately and effectively measures learning outcomes will need to go beyond traditional evaluation concepts (the Kirkpatrick model) and traditional evaluation data collection approaches. Departments and agencies must be involved in data collection. In short, a more comprehensive and strategic approach is required to meet the demands of departments and agencies, as well as the needs of the School, by which methodologies are adopted in line with evaluation needs.

    Recommendation 4. It is recommended that the School builds upon existing learning evaluation practices, and invests appropriately to create a comprehensive evaluation regime capable of collecting, analyzing, and reporting on data in relation to satisfaction, learning, application and other concepts according to the needs of users.

  5. SDI has given the School an upgraded business intelligence system capable of generating a wide range of reports; internal reports have been well received, however, client organizations remain unsatisfied. As a result of the activities undertaken as part of the Initiative, the School has invested in an upgraded business intelligence system that, combined with existing data collection practices, is capable of providing a wide range of outcome metrics. School officials are generally satisfied with reports generated for internal use. Various iterations of external report models, however, have yet to fully satisfy the needs of client departments and agencies. Department and agency officials want monthly usage figures, information on completions and cancellations, and learning outcome metrics for their employees including statistical information such as response rates.

    Recommendation 5. It is recommended that the School improve its external reporting by establishing a system that generates reports that meet client requirements in terms of frequency, content, and format.

Management response and action plan

Management response and action plan. Read down the first column for the recommendation number (from 5.1 to 5.5), then to the right for the recommendation, followed by the management response, the actions to be taken, the timelines, and the offices of primary interest.
No. Recommendation Management response Actions Timelines OPI
5.1

It is recommended that, in future, estimated costs associated with the development and delivery of learning become a distinct element of consideration in Treasury Board submissions and Cabinet requests.

Agreed.

CSPS will work with TBS to address this recommendation.

CSPS will work with TBS to develop a resourcing approach to be followed in instances when submissions have enterprise-wide learning impacts beyond the absorptive capacity of CSPS.

April 2018

Corporate Services Branch, Chief Financial Officer

CSPS will work with departmental (OGD) leads to ensure its costs are factored where proposals require dedicated offerings by CSPS.

On an as and when required basis

Learning Programs Branch, Vice-President
5.2

It is recommended that the School, in consultation with senior officials as well as middle managers and front-line staff across all regions, continues to clarify, validate and refine the definition of its core common curriculum.

Agreed.

CSPS will seek annual re-validation of the core curriculum framework through existing Governance (CSPS DM Advisory Committee).

CSPS received approval in June 2017 of the core curriculum framework by the CSPS DM Advisory Committee.

June 2017

Learning Programs Branch, DG, Curriculum Management, Policy and Strategic Relationships

Consultations to be undertaken with middle managers and regional employees to ascertain whether refinements are required.

April 2018

The core curriculum framework will be presented each year to CSPS Governance (CSPS DM Advisory Committee) to ascertain whether refinements are required.

June 2018
5.3

It is recommended that the School continues to develop and optimize the range of learning modes associated with its learning ecosystem.

Agreed.

CSPS will use data analytics, client feedback and review of learning methodologies to maximize the use of learning modes to achieve learning objectives.

CSPS will increase its use of learner-specific data and business intelligence by developing a personalized learner strategy.

June 2018

Learning Programs Branch, DG, Communications, GCcampus and Learning Solutions

CSPS will develop a plan to maximize the use of learning modes.

September 2018

CSPS will develop a 2-year roadmap and implementation plan for continued evolution of GCcampus to enhance functionalities and user experience.

June 2018
5.4

It is recommended that the School builds upon existing learning evaluation practices, and invests appropriately to create a comprehensive evaluation regime capable of collecting, analyzing, and reporting on data in relation to satisfaction, learning, application and other concepts according to the needs of users.

Agreed.

CSPS will leverage data analytics and systems in creating a renewed evaluation regime.

CSPS will present a comprehensive plan to strengthen evaluation and reporting functions, including new data analytics tools and systems, to CSPS Governance for review. Priority to be given to ensuring robust analysis is in place to support Departmental Results Framework indicators as approved by the Treasury Board.

Implementation to be completed in 2018–19

Corporate Services Branch, DG, Results and Accountability
5.5

It is recommended that the School improve its external reporting by establishing a system that generates reports that meet client requirements in terms of frequency, content, and format.

Agreed.

CSPS will deliver a system that meets external client reporting requirements.

Departmental Learning Reports have been piloted and shared with client organizations for feedback. CSPS will consult with client organizations to refine the Departmental Learning Reports to ensure they meet client needs. CSPS Governance (ADM/DM Advisory Committees) to be engaged. New departmental learning reports to be rolled out to respond to feedback.

Spring 2018

Corporate Services Branch, Chief Information Officer

Client data to be made available to departments on a quarterly basis.

Phased roll-out beginning of September 2018
Date modified: