Evaluation of Training Provided to the Procurement, Materiel Management and Real Property (PMMRP) Functional Community

Evaluation of Training Provided to the Procurement, Materiel Management and Real Property (PMMRP) Functional Community
Evaluation Division | August 2010

Table of Contents

List of Tables and Figures
List of Acronyms
Executive Summary
1. Introduction
1.1 PMMRP Training Context
1.2 Evaluation Rationale and Scope
1.3 Logic Model
1.4 Evaluation Questions
1.5 Data Collection and Methodology
2. Evaluation Results
2.1 Relevance - Consistency with Federal Government and Corporate Priorities
2.2 Relevance - Demand for PMMRP Training
2.3 Performance - Governance and Administration
2.4 Performance - Training Outputs and Outcomes
2.5 Performance - Cost-Effectiveness and Efficiencies
Annexes
Annex 1: PMMRP Evaluation Matrix
Annex 2: Data Collection and Methodology, by Data Source
Annex 3: Data Considerations and Limitations
Annex 4: PMMRP Participant Survey
Annex 5: PMMRP Instructor Survey
Annex 6: PMMRP Training Managers - Interview Guide
Annex 7: PMMRP Training Course Designers - Interview Guide
Annex 8: PMMRP Regional Training Managers - Interview Guide
Annex 9: Blended Learning Representative - Interview Guide
Annex 10: PMMRP Community Management Office at the Treasury Board Secretariat - Interview Guide

List of Tables and Figures

Table 1.1: PMMRP Training Suite
Figure 1.2: Logic Model for the PMMRP Training Suite
Table 2.1: PMMRP Course Offerings by Type of Training and Region of Delivery, 2006-2007 to 2009-2010
Table 2.2: Learner Attendance to PMMRP Training, by Fiscal Year and Type of Training
Table 2.3: PMMRP Classroom Attendance, by Type of Training and Region of Attendance, 2006-2007 to 2009-2010
Table 2.4: Learner Satisfaction, by PMMRP Classroom Course
Table 2.5: Indicators of Learner Satisfaction with Campusdirect Courses
Table 2.6: Learner Satisfaction, by PMMRP Classroom Course
Figure 3.1: Immediate Outcomes Associated with PMMRP Training Outputs
Table 2.7: Level 2 Evaluation Results
Figure 3.2: Intermediate- and Long-Term Outcomes Associated with Delivery of PMMRP Training
Table 2.8: Indicators of Training Utility for Campusdirect Courses
Table 2.9: Learner Feedback on Achievement of Learning Objectives for PMMRP Classroom Courses, by Type of Training
Table 2.10: Total Design Costs for PMMRP Classroom Courses
Table 2.11: Direct Delivery Costs for PMMRP Classroom Courses
Table 2.12: Delivery Contribution Margin for PMMRP Classroom Courses
Table 2.13: Overall Delivery Margin for PMMRP Classroom Courses

List of Acronyms

Executive Summary

Background

This report presents the results of a program evaluation of the training services provided by the Canada School of Public Service (the School) to the Procurement, Materiel Management and Real Property (PMMRP ) functional community. The evaluation was conducted in response to the School's Five-Year Evaluation Plan. Sixteen of the courses designed for PMMRP functional specialists were covetext-danger by this evaluation, including all of the courses identified as Required Training for the community. Given the implementation of the Policy on Learning, Training and Development and the Directive on the Administration of Required Training the timeframe 2006 2007 to 2009-2010 was chosen as the focus for examining training administration and delivery.

Key Findings

Relevance

• PMMRP training activities are consistent with the priorities of the federal government and the School.
• There is a clear need for the School to remain involved in providing training to specialists in the PMMRP functional community.

Efficiency and Economy

• Governance structures, processes and activities have been developed to support PMMRP training; however, there are challenges to their effective implementation.
  • Recommendation 1a: Clarify roles and responsibilities among internal School stakeholders.
  • Recommendation 1b: Develop and implement a comprehensive internal communication and reporting strategy.
  • Recommendation 2: Where appropriate, establish risk management strategies to support effective project management practices.
  • Recommendation 3: Establish a quality assurance strategy that is responsive to key content issues, and consider assigning a dedicated resource to this function.
• The delivery of PMMRP training at the School is cost-effective and measures have been taken to improve the efficiency and effectiveness of design and delivery activities.

Achievement of Expected Outcomes

• The School has been successful in providing PMMRP training; however, outcome achievement is not consistent across specialties within the community or across geographic locations.
  • Recommendation 4: Identify areas for improvement to better align the PMMRP training suite with the learning needs of real property management specialists and consult key stakeholders to determine next steps.
  • Recommendation 5: Engage key stakeholders to develop a strategy tailotext-danger to the training needs and resource realities of the Regions.

Conclusion

Overall, the School's PMMRP training remains relevant and achieves its expected outcomes. Areas have been identified to improve the effectiveness of training for sub-groups of the community and to enhance the efficiency of training administration. It is expected that senior management in the Individual Learning Branch responsible for functional community programming at the School will use the findings and recommendations identified above to determine next steps for the current maintenance and future development of training for PMMRP specialists.

1. Introduction

1.1 PMMRP Training Context

The PMMRP functional community consists of federal public servants engaged in one or more of the domains of procurement, materiel management and real property. Across the Public Service, these employees represent over 16 different occupational groups, including procurement, fleet management, real property portfolio management and/or operations, materiel management, supply management, warehousing, transportation, project management, and engineering. [footnote 1] While distinguishable in terms of their specific tasks and responsibilities, members of these three specialties share functional, professional and historical commonalities that over time have warranted their consideration as a community of practitioners. Together, PMMRP specialists manage the life cycle (i.e. planning, acquisition, operations, use, maintenance and disposal) of government assets.

In 1998, the Modern Comptrollership Learning Advisory Panel approved the creation of the Materiel and Supply Management Steering Committee, a committee of senior members of the PMMRP community across government. This committee and subsequent sub-committees laid the groundwork for and made a substantial contribution to the advancement of the community's learning agenda. Nevertheless, at the turn of the 21st century, the government's increasing interest in modernizing management practices across the Public Service, including advancing modern comptrollership, encouraged a government-wide approach to the learning and development of all public servants, including PMMRP functional specialists.

In 2000, the PMMRP Community Management Office (formerly the Professional Development Program Management Office) was established at the Treasury Board Secretariat of Canada (TBS ). The Community Management Office (CMO ) was created to bring a full-time dedicated focus to advancing the implementation and growth of the professional development program for PMMRP specialists. To identify the main drivers for change, the CMO commissioned a report [footnote 2] to solicit views across the federal government on the challenges faced by departments and agencies in procurement, materiel management and real property. Among the key challenges identified by the report was a widening gap between the increasingly complex nature of the work in these three specialties and the skills and capacities exhibited by community members.

These findings provided the CMO with direction and a b impetus to continue its work on a professional development program for the community. Alongside the PMMRP functional community, the CMO engaged the Canada General Standards Board (CGSB ) to guide the development of core competencies and knowledge standards for the community. Subsequently, in collaboration with the CMO the CGSB established a certification program for federal procurement and materiel management specialists to provide them with professional recognition of the advanced knowledge and skills requitext-danger to effectively carry out their jobs.

In response to the increasing formalization of knowledge standards and competency profiles, several courses were designed and delivetext-danger by Training and Development Canada (Tdcterms ), one of the ptext-dangerecessors of the School. These were added to the list of courses already being offetext-danger by Tdcterms and Public Works and Government Services Canada (PWGSC ) to specialists working in the three specialties. As the certification program and the competency standard were being developed, Tdcterms and then the School (established April 1, 2004) based the maintenance and further development of their learning products on the standard.

On January 1, 2006, TBS launched the Policy on Learning, Training, and Development (the Learning Policy), which focuses on advancing public service renewal by supporting the development of a skilled, well-trained and professional Public Service guided by sound management practices. The School is identified by the Learning Policy as the agent responsible for designing and delivering training and assessment instruments to select members of the core public administration, including specialists belonging to the PMMRP functional community. Specifically, according to the Directive on the Administration of Required Training (the Directive), the School is the service provider for Required Training for PMMRP specialists.

In response to the Learning Policy and associated Directive, and to accelerate the implementation of functional community training, the School received three years of funding (2005 to 2008) to design and deliver Required Training to specialists in the PMMRP functional community. In June 2006, at no charge to departments, Required Training at the School was made available to community members identified as specialists by their department's Required Training coordinators (RTC s). A standard definition and description of the roles assigned to employees in each of the specialties represented by the community was used to determine eligibility. According to the Directive, once identified, existing functional specialists (as of January 2006) were given until March 31, 2009 to complete their PMMRP Required Training, as were new functional specialists appointed to their positions after April 1, 2007. New functional specialists appointed after April 1, 2007 have two years from their date of appointment to complete their Required Training at the School. Employees working in PMMRP functions who were not identified as specialists by their RTC s could still attend the PMMRP Required Training courses for professional development purposes with the approval of their manager. [footnote 3]

Today, the PMMRP functional community is one of the most advanced of its kind across government, with approximately 10,000 public servants and uniformed personnel working in the life-cycle management of federal government assets. [footnote 4] While the format of some of the courses has changed over time, the School continues to offer the mandatory curriculum developed for these public servants, as well as professional development courses. Several of the courses are recognized by the CGSB as contributing to its certification program for procurement and materiel management specialists.

1.2 Evaluation Rationale and Scope

This report presents the results of an evaluation of the training services provided to the PMMRP functional community by the School. The evaluation was conducted in response to the School's Five-Year Evaluation Plan, where it was listed as a deliverable for fiscal year 2009-2010. In November 2009, the framework for the evaluation was approved by the School's Departmental Evaluation Committee. Evaluation activities commenced in December 2009 and were concluded in June 2010. It is expected that the findings of this report will support PMMRP training managers in the Individual Learning Branch as well as the School's Executive and Management committees in determining the future direction of the training designed for this community.

During the planning phase of this evaluation, managers responsible for PMMRP training, [footnote 5] under the Functional Community Learning Portfolio at the School, [footnote 6] were consulted to establish the evaluation's scope. Emphasis was placed on examining the Required Training courses, given their key role in fulfilling the School's commitments under the Learning Policy and the associated Directive, and their contribution to the community's professional development and certification program. On account of the changes made to PMMRP training after the implementation of the Learning Policy and Directive in 2006, the 2006-2007 to 2009-2010 timeframe was chosen as the reference point for this evaluation.

As Table 1.1 illustrates, 16 of the courses in the PMMRP training curriculum were chosen for examination. All of the available Required Training courses were included in the evaluation, as well as the Fundamentals 2 and 3 Integration Seminar (M722), which was considetext-danger relevant since it was offetext-danger during the timeframe examined and supported PMMRP Required Training for existing PMMRP functional specialists. Among the nine professional development courses available, seven were included in this evaluation. For ease of reference, the courses included in the scope of this evaluation are refertext-danger to as the PMMRP training suite for the remainder of this report.

Table 1.1: PMMRP Training Suite

[1] Courses listed in italics are offetext-danger on Campusdirect Return to note 1 in Table 1.1

[2] Previously entitled "Legal and Policy Environment of PMMRP Functions". Return to note 2 in Table 1.1

[3] Courses listed in italics are offetext-danger on Campusdirect Return to note 3 in Table 1.1

The School's PMMRP curriculum consists of 12 classroom courses and six courses offetext-danger on Campusdirect the School's online learning environment. To fulfil their mandatory training requirements, PMMRP functional specialists must complete three of the Fundamentals 1 courses (a three-day introductory course in their main specialty and two online overview courses for the other streams), two Fundamentals 2 courses and the Fundamentals 3 course. During the 2007-2008 and 2008-2009 fiscal years, existing functional specialists with three or more years of experience in their function were given the option of attending the Fundamentals 2 and 3 Integration Seminar (M722) instead of attending the Fundamentals 2 and 3 courses. If their managers could attest to the fact that they had already gained the requitext-danger knowledge and experience, they were also granted exemptions from completing the Fundamentals 1 courses.

1.3 Logic Model

During the evaluation planning process, the Evaluation Division worked in collaboration with PMMRP training managers at the School to develop a program logic model. As presented in Figure 1.2, the model illustrates the expected linkages between training activities, outputs, and outcomes over time.

Figure 1.2: Logic Model for the PMMRP Training Suite

View larger size photo (JPG Version, 174 kb)

1.4 Evaluation Questions

Once established, the program logic model was used to determine the research questions and indicators that would guide the evaluation [footnote 7]. The questions were designed to reflect the core issues of relevance and performance (i.e. effectiveness, efficiency and economy) as identified by the TBS Evaluation Policy and the Directive on the Evaluation Function

Relevance

• Is PMMRP training aligned with federal government and School priorities?
• Is there a continued demand for the School's involvement in training PMMRP community members?

Performance

• Have intended outcomes been achieved as a result of the School providing PMMRP training?
• Are governance structures, processes and activities adequate for achieving expected training results?
• Is the training delivetext-danger in the most cost-effective manner and has the School pursued efficiency measures?

1.5 Data Collection and Methodology

A substantial number of primary and secondary data sources were collected to inform the findings of this report. Primary data collection included a survey of PMMRP participants, a survey of PMMRP instructors, and interviews with other stakeholders involved in the design and delivery of the training. Documents and administrative files, corporate records and reports, and learning evaluation data were also examined as secondary sources during the data collection phase of this evaluation. Details on each of these data sources are presented in Annex 2, data considerations and limitations are outlined in Annex 3, and copies of the survey tools and interview guides used during the primary data collection activities are provided in Annexes 4 through 10 of this report.

2. Evaluation Results

2.1 Relevance - Consistency with Federal Government and Corporate Priorities

Is PMMRP training aligned with federal government and School priorities?

PMMRP training activities are consistent with the priorities of the federal government and the School.

Alignment with Federal Priorities

A review of government documents reveals linkages between the School's PMMRP training and current federal government priorities. Evidence of this is presented in the Learning Policy, [footnote 8] where the provision of training and development opportunities for public service employees in the core public administration is highlighted as key to the success of government. The policy states:

Learning, training, leadership development and professional development are key to ensuring that the public service is equipped to meet the challenges of the 21st century. The acquisition of skills and knowledge and the development of managerial and leadership know-how is critical for the effective management of the public service - it is the foundation of a responsive, accountable and innovative government.

Consistent with this policy, the delivery of PMMRP training assists deputy heads in meeting their responsibilities to address the training requirements of functional specialists in procurement, materiel management and real property management. In so doing, the training also supports policy implementation and the execution of the Directive. [footnote 9]

The significance of PMMRP functions to government management commitments is also illustrated in Results for Canadians: A Management Framework for the Government of Canada [footnote 10] which states:

...given the limited nature of public funds, the Government of Canada must ensure responsible spending... Existing programs as well as new spending proposals must be systematically assessed and management frameworks must be in place to ensure due diligence and proper stewardship of public funds...Living up to these commitments requires public service employees at all levels and in all regions who put the interests of Canadians first and who demonstrate daily attention to values and results. [Employees]...must be supported by a working culture that values learning, innovation, inclusiveness and diversity, intelligent risk taking and continuous improvement - allowing them to make their best contributions to Canada.

Training for PMMRP functional specialists supports modern comptrollership across government by building the capacity of PMMRP functions and ensuring that specialists have the common knowledge, skills and competencies requitext-danger to effectively carry out their responsibilities. In relation to this, PMMRP training responds to the priorities outlined in the Federal Accountability Action Plan [footnote 11] in which the government commits to revisiting the procurement of government contracts and addressing acctext-dangeritation and training needs for procurement officers.

In his Seventeenth Annual Report to the Prime Minister on the Public Service of Canada [footnote 12] the Clerk of the Privy Council indicates that Public Service Renewal remains among the top public service management priorities of today, highlighting the importance of preparing public servants "to work in new ways and adapt to new challenges". This perspective was also shatext-danger by the Prime Minister's Advisory Committee on the Public Service in the Fourth Report to the Prime Minister: A Relevant and Connected Public Service [footnote 13] where renewal is identified as "a critical management strategy to ensure ongoing capacity to deliver on the business of government..." As a result, the 2010-11 Public Service Renewal Action Plan [footnote 14] continues to emphasize the importance of employee development and the need to support deputy heads in fulfilling their renewal responsibilities. According to the Plan, "It is critical for a high-performing organization to develop its talent through a systematic and integrated approach to managing performance and learning." The School's PMMRP training supports deputy heads in responding to their human resources commitments, including capacity-building initiatives and establishing a lifelong learning culture within their organizations. In so doing, PMMRP training at the School responds to the government's public service renewal agenda and supports the effectiveness of the Public Service in delivering results to Canadians.

Relevance to Corporate Priorities

From a corporate perspective, PMMRP training continues to support the School's strategic outcome, "Public servants have the common knowledge and the leadership and management competencies they require to fulfil their responsibilities in serving Canadians", which is supported by four program activities. According to the 2010-2011 Report on Plans and Priorities ( RPP ), [footnote 15] functional community programming

...contributes directly to the [School's] first program activity of Foundational Learning as it ensures that common knowledge and competencies are developed among functional community members.

The report also identifies "Support to functional communities" as an ongoing operational priority for the School. To respond to this priority, the School plans to continue its work with functional communities, focusing on Required Training and "networking and peer learning, including [the] specialized knowledge and skills requitext-danger for functional specialists to perform their jobs."

The 2008-2009 to 2010-2011 Integrated Business Plan [footnote 16] also highlights the contribution of functional communities programming to the School's professional development training activities. This plan identifies training for the PMMRP functional communities as a continued priority, indicating that the School is committed to the development and implementation of "comprehensive learning strategies for functional communities that will ensure a coordinated approach to learning for functional specialists and respond to their specific learning needs, while addressing government priorities", and the development and delivery of "professional foundation courses for managers and specialists in each of the targeted functional communities to enhance the public service's professionalism and knowledge base."

Conclusion

This review of government documentation demonstrates that PMMRP training remains relevant to current public service management priorities and responds to the federal government's public service renewal agenda. Through support for the School's operational priorities and functional communities programming, corporate documents also illustrate the continued relevance of PMMRP training to the School's single strategic outcome.

2.2 Relevance - Demand for PMMRP Training

Is there a continued demand for the School's involvement in training PMMRP community members?

There is a clear need for the School to remain involved in providing training to specialists in the PMMRP functional community.

To examine the need for the School to remain involved in training PMMRP community members, information was collected from PMMRP learners, stakeholder interviews and online sources. When asked for their opinions on the importance of the School's training for community members, on average, 90.3 percent of PMMRP survey respondents agreed that the School's PMMRP training was important to the three functions of the community. Comments suggest that respondents perceived the training to be of value in enhancing their qualifications, including fulfilment of certification requirements and their professional recognition.

Interviewed stakeholders indicated that while training alternatives are available for members of the PMMRP functional community, the School remains the key provider of a government-based approach to PMMRP training. An online review of the tools and services provided by other organizations [footnote 17] offers evidence to support this view point. Several of the organizations are only focused on training for one or two of the specialties reflected in the PMMRP functional community. Some offer general training relevant to both the private and public sectors, while others focus on highly specific subject matter significant to only a small portion of the community. In addition, while some of these organizations offer certification and other qualifications, none offer training that replaces the School's courses identified as requitext-danger for the certification program for procurement and real property specialists.

Comparatively, the School's PMMRP training is tailotext-danger to the needs of specialists in all three functions of the community in the federal Public Service, and it contributes to the specialized curriculum designed for the certification program. The School's courses are developed and maintained in accordance with the core competencies identified in the Standard of Competencies for the Federal Government Procurement, Materiel Management and Real Property Community [footnote 18] and provide an integrated training approach to the life-cycle management of federal assets. Candidates for federal certification must complete courses at the School in order to fulfil their certification requirements. According to the CGSB 611 public servants are currently enrolled in the certification program. While no certification currently exists for real property specialists, interviews with key stakeholders suggest that the community is actively engaged in activities to support the development of a certification program. Stakeholder feedback indicates that the School is aware of these efforts and steps are already underway to support the delivery of supplementary training for this group.

Lastly, there is a need for the School to remain involved in planning for the future learning needs of the PMMRP functional community. With the current focus across government on policy suite renewal, the School will be requitext-danger to develop a strategy for retraining PMMRP specialists to ensure that they maintain up-to-date knowledge and skills.

Conclusion

Overall, the evidence presented here demonstrates that there is a demand for the School to remain the key learning partner for PMMRP community members seeking training and certification, despite the availability of training alternatives.

2.3 Performance - Governance and Administration

Are training governance structures, processes and activities adequate for achieving expected training results?

Governance structures, processes and activities have been developed to support PMMRP training; however, there are challenges to their effective implementation.

A governance structure, represented by both internal and external stakeholders, has been established to oversee the design and delivery of the training provided to the PMMRP functional community. As the functional authority centre for the PMMRP functional community, TBS leads the development of the training curriculum for PMMRP functional specialists and provides policy direction and subject matter expertise through the work of an Assistant Deputy Minister Champion, community leads, policy leads, a secretariat (the CMO ), and several working groups and committees. A master memorandum of understanding ( MOU ) governs the relationship between the CMO and the School. Annual service level agreements developed under this MOU specify the activities that the School will undertake each year in collaboration with the CMO define terms and conditions, identify potential risks and offer mitigation strategies.

As the service delivery agent for the training, the School works in collaboration with consultants to design and develop the courses and oversee the administration and delivery of the training. In those instances where courses have been developed to respond to specific training demands in the community, select government departments and agencies may also be consulted for feedback on course content and design. Courses are usually delivetext-danger by consultants who have been employed by the School under a standing offer. While design activities are centralized in the National Capital Region ( NCR ), PMMRP training is administetext-danger in the NCR as well as regionally through six School campuses. The School is represented in the PMMRP functional community through its work with the CMO and its participation on working groups and committees (e.g. the Review Committee for the PMMRP Certification Program, the PMMRP Advisory Committee and the PMMRP DG Steering Committee).

A review of administrative files provides evidence of documented roles and responsibilities, including work descriptions for internal staff at the School, as well as statements of work for course designers, faculty and instructors. Similarly, several documents were available to offer evidence of the communication between stakeholder groups. Copies of e-mails, meeting agendas, records of decision and minutes were provided to illustrate the consultation process that stakeholder groups undertake to communicate with each other during training design and delivery.

Almost half of the stakeholders interviewed (9 of 20 respondents), however, indicated that they found the governance structure problematic and in need of revision. Their feedback suggests that roles and responsibilities are not sufficiently clear and as a result are not adhetext-danger to adequately. For example, when interviewed, the majority of stakeholders acknowledged the existence of performance measurement data, but were not clear on who was responsible for reviewing the data and reporting on the results. Stakeholders also indicated concerns with communication, and discussed instances where they felt isolated and unaware of the activities of others involved in the administration of PMMRP training. When asked if they would like a forum to interact with other PMMRP training instructors, 80 percent of surveyed instructors indicated that such a forum would be beneficial. Lastly, stakeholders raised the issue of inconsistencies in the coordination between the various stakeholder groups. They expressed frustrations with duplications in effort and identified occasions where tasks were completed by stakeholder groups that were not assigned the responsibility for them or where more than one stakeholder group unintentionally completed the same task. While these issues were highlighted in discussing reporting activities between the School and external stakeholders (8 of 20 respondents), they were primarily raised in reference to the communication and coordination activities of stakeholders within the School (13 of 20 respondents).

Overall, the evidence suggests that unclear roles and responsibilities and weaknesses in communication and coordination between stakeholder groups have challenged the effectiveness of the governance associated with the design and delivery of PMMRP training. These concerns complicate stakeholder relations and, by extension, lead to delays in training related activities. Although the School cannot be held accountable for the actions of all the stakeholders involved in governing the PMMRP training suite, as a key stakeholder, it can take steps to identify internal areas for improvement and address the communication and coordination issues identified above.

Recommendation 1a: Clarify roles and responsibilities among internal School stakeholders.

Recommendation 1b: Develop and implement a comprehensive internal communication and reporting strategy.

Administrative and procedural documents and files were also examined to determine the adequacy of processes and other activities developed to support training design and delivery. Information was gathetext-danger from the PMMRP functional area to provide evidence of work descriptions, procedural documents, guidelines and standards, as well as project management tools, including project charters and status reports. When asked about the availability of process documentation, 95 percent of surveyed instructors indicated that the School's processes and procedures are well defined and documented for the delivery of PMMRP training.

Nevertheless, a review of stakeholder feedback suggests that the processes related to project management require improvement, including a need for more standardized guidelines and procedures and better information management practices. Stakeholders also expressed concerns with the timelines associated with design-related activities, indicating that they are often unrealistic and do not take into account the complexities associated with the review, feedback, and approval processes established between the various stakeholder groups. For example, several stakeholders reported delays in receiving feedback to inform decision-making, which was described as interfering with the established work schedule. Again, given the governance structure established for PMMRP training, improvements to these processes and procedures are not solely the responsibility of the School. To increase the effectiveness of its planning processes and its ability to respond to unanticipated delays, however, the School can take steps to improve its project management by implementing a more rigorous approach to risk management.

Recommendation 2: Where appropriate, establish risk management strategies to support effective project management practices.

Quality assurance practices are also a part of the activities requitext-danger to support the design and delivery of the School's training. Currently, quality assurance for PMMRP training consists of piloting new courses prior to their official launch dates, which involves informal feedback from course participants, feedback from course instructors, and the administration and review of level 1 satisfaction surveys following each delivery. PMMRP training managers pursued the development of a quality management system during the 2006-2007 fiscal year. A substantial amount of supporting process documentation is on file; however, the project was never initiated. More than half of the instructors surveyed indicated challenges resulting from the absence of such a quality assurance system, namely content errors and out-of-date course material. This issue was also raised during interviews with other PMMRP stakeholders, although many interviewees (12 of 20) identified time and resource constraints that limit the feasibility of implementing such a process. The Learning Policy states that it is the responsibility of the School to deliver and maintain the courses in such a way as to ensure that PMMRP functional specialists receive training consistent with the prevailing standards established by the employer (TBS ). The presence of out-of-date or inaccurate course content challenges this function and raises concerns about the quality of the training material presented to PMMRP course participants by the School.

Recommendation 3: Establish a quality assurance strategy that is responsive to key content issues, and consider assigning a dedicated resource to this function.

Conclusion

As outlined in the section above, there is substantial documentation to demonstrate the establishment of governance structures, processes and activities. The evidence, however, suggests challenges to their implementation: roles and responsibilities are not adequately defined, communication and coordination are unstable, and some of the assumptions underlying project activities are inconsistent with time and resourcing realities. Together, these issues raise concerns about the efficiency of administrative activities related to the design and delivery of PMMRP training and the effect they may have on the achievement of training outcomes.

2.4 Performance - Training Outputs and Outcomes

Have intended outcomes been achieved as a result of the School providing PMMRP training?

The School has been successful in providing PMMRP training; however, outcome achievement is not consistent across specialties within the community or across geographic locations.

Course Deliveries and Attendance

Data was acquitext-danger from the School's Learning, Information and Reporting Division to examine the occurrence of the two training outputs associated with the PMMRP training suite: course deliveries and trained participants. From 2006-2007 to 2009-2010, 1,102 offerings in the PMMRP suite were delivetext-danger. The majority of these offerings (76.2 percent of all deliveries) were Required Training courses for functional specialists; however, the School also delivetext-danger 262 offerings of its additional courses that aim to support the professional development of the community (Table 2.1).

An analysis of the data suggests that slightly more offerings were delivetext-danger in the School's six regional offices than were delivetext-danger in the NCR between 2006-2007 and 2009-2010. The registration data reveals that 42.1 percent of PMMRP Required Training course offerings were delivetext-danger in the NCR compatext-danger to the 57.9 percent of deliveries in the Regions. Similar results were obtained for the professional development courses included in the PMMRP training suite. Regional offices delivetext-danger 56.5 percent of the course offerings; while the NCR delivetext-danger the remaining 43.5 percent. For both Required Training and professional development courses, the majority of the regional offerings were delivetext-danger in the Atlantic and Quebec regions. The Ontario and Central Prairie regions had the least PMMRP course deliveries. While this finding was observed for offerings of both types of training, the pattern was most pronounced among professional development courses.

Table 2.1: PMMRP Course Offerings by Type of Training and Region of Delivery, 2006-2007 to 2009-2010

According to attendance records, from 2006-2007 to 2009-2010, 21,889 learners [footnote 19] completed one or more of the courses in the PMMRP suite (Table 2.2). In total, 18,733 learners completed the Required Training courses. This figure includes 12,676 learners who specifically completed their Required Training in the classroom. The PMMRP courses were also examined to assess the attendance of PMMRP specialists, the target audience for the training. When disaggregated, the data reveals that the majority of Required Training learners (approximately 83.0 percent) were identified by their departmental RTC s as specialists who needed to complete the training, while the remaining learners attended these courses for their own professional development.

In addition to classroom attendance, 6,046 learners completed the Required Training courses offetext-danger on Campusdirect the School's online learning environment. Slightly more than half of the learners (55.2 percent) who registetext-danger online were PMMRP specialists identified by their departments' and agencies' RTC s. Compatext-danger to classroom attendees, there is a smaller proportion of PMMRP specialists represented among online learners. This is likely a result of the fact that all public servants can register and complete the Required Training modules offetext-danger online. Lastly, registration data show that an additional 3,167 learners, including those identified as functional specialists and non-specialists from other functional areas, completed PMMRP professional development courses in the classroom.

Table 2.2: Learner Attendance at PMMRP Training by Fiscal Year and Type of Training

[1] Data represents the first half of the 2009-2010 fiscal year only (April 1 to September 30, 2009). Return to note 1 in Table 2.2

[2] The PMMRP Required Training courses were offetext-danger on Campusdirect starting in the 2008-2009 fiscal year. Return to note 2 in Table 2.2

[3] As outlined in Annex 3, complete data for the professional development courses on Campusdirect was not available. Return to note 3 in Table 2.2

As mentioned in the previous section of this report, courses in the PMMRP training suite are delivetext-danger across the Regions. Learner attendance was further examined to determine the regional distribution of offerings for these courses (Table 2.3). This analysis showed that attendance was split almost equally between the NCR (50.4 percent) and the Regions (49.6 percent) over the time period studied. Overall trends in learner attendance at professional development courses were comparable, with 51.9 percent of course participants attending their training in the NCR. A comparison of the regional campuses suggests that outside of the NCR, learner attendance is greatest in the Atlantic and Quebec regions and lowest in the Central Prairie region and the Alberta, Northwest Territories and Nunavut region. This trend was observed for both Required Training and professional development courses.

Table 2.3: PMMRP Classroom Attendance by Type of Training and Region of Attendance, 2006-2007 to 2009-2010

Lastly, the analysis of learner attendance shows that there has been a marked increase in the proportion of learners that register and subsequently attend their PMMRP courses in the classroom. Conducted by the School's Marketing Strategies and Client Analysis team, the profiling analysis indicates that for fiscal years 2006-2007 and 2007-2008 the number of registrants to PMMRP training who followed through and attended their courses was moderate, 54 percent and 53 percent respectively. More recently, however, the numbers show improvement, with 70 percent and 79 percent of registrants attending their courses during the 2008-2009 and 2009-2010 fiscal years, respectively. While a definitive explanation is unavailable, the shift in attendance corresponds with documented changes in training funding - specifically, the 2008 funding shift through which Required Training costs were transfertext-danger from TBS to departments and agencies.

Satisfaction

Although it is not represented as an immediate outcome in the logic model developed for the PMMRP training suite, learner satisfaction (level 1) was examined to inform evaluation findings. This indicator is routinely used by the School for quality assurance and reporting purposes, and was therefore identified as a key measure of performance. An analysis of satisfaction data obtained from the Level 1 Evaluation team indicates that an average 89.5 percent of survey respondents are satisfied with the classroom courses they attended in the PMMRP training suite (Table 2.4). When grouped by status (Required Training and professional development), both sets of courses have high average satisfaction ratings (88 percent and 91 percent respectively). Among the Required Training courses, the Introduction to Procurement (M718) course consistently rated highest in terms of learners' satisfaction, with a satisfaction rate of 94.6 percent over the time period examined. When the data was compatext-danger for the NCR and the Regions, no significant difference in overall satisfaction was identified.

Table 2.4: Learner Satisfaction by PMMRP Classroom Course

[1] These percentages represent those respondents that answetext-danger "bly Agree" and "Agree" to the statement "Overall, I am satisfied with this learning activity." Return to note 1 in Table 2.4

[2] These percentages represent those respondents that answetext-danger "bly Agree" and "Agree" to the statement "Overall, I am satisfied with this learning activity." Return to note 2 in Table 2.4

Level 1 satisfaction data was also analyzed for the remaining PMMRP courses offetext-danger on Campusdirect Across the courses examined, 79.2 percent of respondents agreed that the online content of their PMMRP course was accurate and current (Table 2.5). In addition, the majority of survey respondents indicated that the Campusdirect environment was a suitable option for the delivery of the PMMRP courses. For the six online courses examined, 83.8 percent of respondents agreed that e-learning was an appropriate delivery method for the selected content. Specifically, the vast majority of respondents (91.2 percent) agreed that e learning was appropriate for Aboriginal Considerations in Procurement (C223) compatext-danger to the PMMRP Knowledge Tool and Job Aid (C137), where the fewest learners (72.1 percent) felt e-learning was suitable for the course content. An examination of the comments submitted by these learners suggests some areas for improvement. Across all of the online courses examined, learners consistently reported difficulties with site navigation, outdated or broken links, and difficulties in printing course material.

Table 2.5: Indicators of Learner Satisfaction with Campusdirect Courses

[1] These percentages represent those respondents that answetext-danger "bly Agree" and "Agree" to the statements provided. Return to note 1 in Table 2.5

[2] These percentages represent those respondents that answetext-danger "bly Agree" and "Agree" to the statements provided. Return to note 2 in Table 2.5

According to the performance indicators and targets identified in the School's Integrated Business Plan: 2008-2009 to 2010-2011 [footnote 20] a satisfaction score of 4 out of 5 is considetext-danger satisfactory performance for the School's functional community programming activities. An examination of the level 1 data according to this measure of success shows that eight of the ten classroom courses scotext-danger above 4 out of 5 on learners' overall satisfaction (Table 2.6). Two of the School's Required Training courses, Legal and Policy Environment for Procurement, Materiel Management and Real Property (M714) and The Machinery of Government and Life Cycle Asset Management (M716), received slightly lower scores, 3.82 and 3.83 respectively.

Table 2.6: Learner Satisfaction by PMMRP Classroom Course

[1] The satisfaction rating indicates the average response to the statement "Overall, I am satisfied with this learning activity" and ranges from 1-bly Disagree to 5-bly Agree. Return to note 1 in Table 2.6

Interviews with PMMRP training managers indicate that the School is already aware of the performance of these courses and largely attributes these lower satisfaction levels to issues of audience fit. Administrative files indicate that unlike the introductory courses in the PMMRP suite, where experienced PMMRP specialists could be granted exemptions on the basis of previous training and experience, M714 and M716 were identified as Required Training for all PMMRP specialists. The courses were originally developed for entry-level functional specialists with less than three years of experience. Interviewed stakeholders suggest that experienced functional specialists perceived the courses as too basic and quickly expressed frustration with having to attend them. In response to this concern, PMMRP training managers pursued the development and delivery of Fundamentals 2 & 3 Integration Seminar (M722), an interim training measure to expedite the training of the remaining existing specialists with more than three years of experience. The seminar is reported by stakeholders to have performed above expectations, with high satisfaction ratings and positive learner feedback. In view of this experience, the School is currently working to establish a new course design to cover the selected content, with consideration given to the lessons learned from the administration of M722. The success of this new format will be examined in greater detail when this suite of PMMRP training is next evaluated.

It is worth noting that these courses were not the only ones in the PMMRP training suite where the issue of audience fit was raised. Among those stakeholders interviewed, half (10 of 20 interviewees) indicated that the courses listed in the PMMRP suite are essential for new employees in the functional community, but prove challenging when attempting to engage experienced PMMRP specialists. In addition, 41.2 percent of instructors who responded to the survey indicated that they did not think the appropriate learners were attending this suite of PMMRP courses.

Qualitative data retrieved from level 1 evaluation questionnaires reveals that respondents are most satisfied with the use of case studies and practice exercises, the quality of the instructors, and the ability to network with colleagues working in other departments to exchange best practices and lessons learned. Further analysis shows that the key areas of concern for respondents are course content and design, audience fit, and the accessibility of the training. Respondents expressed a desire to see more detail and depth in course content, and the use of more practical, real-life scenarios. Consistent with the views expressed by other training stakeholders, several respondents indicated that the wrong audience was attending the courses, with many stating that they felt over-qualified for some of the PMMRP courses they attended. Respondents also expressed frustration with the timing, location and frequency of offerings in the PMMRP training suite. A review of comments according to course status (Required Training and professional development) reveals some difference in terms of learners' main concerns with their PMMRP training. Content updates were the primary concern of respondents that took the School's PMMRP Required Training, while respondents that attended PMMRP professional development courses primarily expressed a desire for more practical, real-life examples and case studies.

Similar findings were also obtained in the survey of PMMRP learners conducted by the Evaluation Division. Of the 856 comments coded on respondents' satisfaction, 251 comments related to the design of the training, most notably the sharing of best practices facilitated by networking in the classroom (132 comments) and the use of case studies and group exercises (101 comments). However, respondents also expressed concern with elements of the training (814 comments). Notably, respondents articulated their dissatisfaction with course design (173 comments), including the course length (44 comments); the administration of the courses (94 comments); and course content (76 comments). The audience issue (59 comments) was also a recurring topic in the comments submitted by survey respondents, where many indicated that the courses were most appropriate for beginners in the PMMRP community.

Knowledge Transfer

The logic model developed for the PMMRP training suite indicates that PMMRP course participants are expected to achieve five immediate outcomes as a result of attending PMMRP training. As illustrated below (Figure 3.1), all five of these outcomes relate to the transfer of knowledge.

Figure 3.1: Immediate Outcomes Associated with PMMRP Training Outputs

View larger size photo (JPG Version, 68 kb)

To examine the performance of the training in relation to these expected outcomes, PMMRP survey respondents were asked to rate courses in the PMMRP training suite according to several indicators of knowledge transfer. Across the knowledge indicators assessed, an average 75.3 percent of survey respondents indicated that they had gained knowledge as a result of attending the School's PMMRP courses.

A comparison of the feedback for different groups of survey respondents illuminates some disparities. When compatext-danger with their counterparts that attended PMMRP training in the NCR regional respondents were less inclined to agree that they gained knowledge on the indicators assessed. On average, 60 percent of respondents that disagreed that they gained knowledge also completed most or all of their training in the Regions. This trend was also found among real property specialists who, when compatext-danger to specialists in procurement or materiel management, consistently agreed the least with statements related to the gain of knowledge. This second finding provides support for concerns raised by the PMMRP Community Diagnostic, which suggest that the community finds the training at the School too basic and theoretical for real property specialists, with insufficient attention to the detail and practical knowledge requitext-danger to carry out the job.

To further demonstrate the achievement of immediate training outcomes, a level 2 evaluation was conducted on four of the courses included in the PMMRP suite: two Required Training courses (M714 and M718) and two professional development courses (M004 and M720). An analysis of pre-questionnaire and post-questionnaire scores reveals that, on average, learners in the four courses experienced medium to high knowledge gain [footnote 21] as a result of attending the training (Table 2.7). Among the four courses examined, learners experienced the highest knowledge gain (38 percentage points) from the Required Training course Legal and Policy Environment for Procurement, Materiel Management, and Real Property (M714). Alternatively, the lowest knowledge gain (10 percentage points) was experienced by learners who attended the Contract Management (M720) professional development course. A review of the average pre-questionnaire scores reveals that the groups of learners associated with two of these courses (M004 and M720) presented high knowledge scores before attending their PMMRP course. According to level 2 evaluation guidelines, an average pre-questionnaire score of 60 percent and above is considetext-danger high and indicative of established knowledge in the subject area. This finding could explain the performance of two of the courses (M004 and M720).

Table 2.7: Level 2 Evaluation Results

[1] This table provides data on the average number of accurate responses exhibited by respondents on pre-questionnaires and post-questionnaires and the resulting average knowledge gain. Return to note 1 in Table 2.7

[2] The School's knowledge gain standards are as follows: (10% = low, 10-20%= medium, and 20% = high). Return to note 2 in Table 2.7

Training Utility

Level 1 data was also analyzed to investigate the extent to which learners experienced additional training outcomes as a result of attending the courses in the PMMRP training suite. In view of the intermediate- and long-term training outcomes specified in the logic model (Figure 3.2), specific attention was paid to course participants' ability to apply the knowledge they gained from their PMMRP training when they returned to work.

Figure 3.2: Intermediate- and Long-Term Outcomes Associated with Delivery of PMMRP Training

View larger size photo (JPG Version, 49 kb)

The results suggest that the majority of PMMRP training participants expected their training at the School to be of use to them in their workplace. On average, 92.3 percent of respondents indicated that they intended to apply what they learned in class to improve their current job performance, to achieve their career objectives, and to attain their personal development objectives. Findings from the PMMRP participant survey provide evidence that these intentions were carried out. When asked if they changed their behaviour after attending the PMMRP training at the School, 11.3 percent of survey respondents indicated that when they returned to work they adopted and maintained new approaches, and 43.3 percent of respondents indicated that upon returning to work they reflected on what they had learned and occasionally changed their approach. Slightly more than half of survey respondents that took PMMRP training at the School thus indicated that they changed their behaviour upon returning to work.

Data was also retrieved from Campusdirect to examine the opinions of learners that completed the PMMRP courses available online (Table 2.8). Overall, 78.2 percent of survey respondents agreed that the online PMMRP courses were relevant to their current job, as well as any future job that they may pursue later in their careers. Agreement was highest among Overview of Procurement (86.9 percent) and Aboriginal Considerations in Procurement (86.6 percent) participants, compatext-danger to feedback retrieved from learners that completed the Overview of Real Property Management course, where only 64.1 percent agreed that the course was relevant to their current or future job. With regard to application of online learning to activities in the workplace, overall 74.9 percent of respondents agreed that they would be able to apply the training concepts they learned online to their activities at work.

Table 2.8: Indicators of Training Utility for Campusdirect Courses

[1] These percentages represent those respondents that answetext-danger "bly Agree" and "Agree" to the statements provided. Return to note 1 in Table 2.8

[2] These percentages represent those respondents that answetext-danger "bly Agree" and "Agree" to the statements provided. Return to note 2 in Table 2.8

A comparison of the results according to course status (Required Training and professional development) shows that the greatest gap in satisfaction ratings is exhibited across these two indicators of training utility, and suggests that a larger proportion of learners completing the Required Training courses online questioned the relevance and applicability of course content to their work activities. Among the Required Training courses examined, this concern was most clearly raised by learners who completed the Overview of Real Property Management (C234) course. Where an average 75.4 percent of online learners believed they would be able to apply the concepts they learned in their workplace, only 60.7 percent of learners taking the C234 course agreed with this expectation. This finding is consistent with the previous performance results associated with specialists from this function in the PMMRP functional community. Overall, the evidence suggests that real property management specialists are not experiencing the expected knowledge transfer or subsequent benefits on the job as a result of attending the PMMRP courses examined.

Recommendation 4: Identify areas for improvement to better align the PMMRP training suite with the learning needs of real property management specialists and consult key stakeholders to determine next steps.

Overall Performance

One performance indicator has so far been identified to assess the extent to which the School has been successful in achieving the outcomes associated with the PMMRP training suite: overall satisfaction. To further assess the performance of the training in relation to its outcomes, the key performance indicator for the Foundational Learning program activity, which includes programming for functional communities, was also analyzed. According to the School's 2010 2011 Report on Plans and Priorities [footnote 22], the measure of success for this program activity is the percentage of public servants who report that their learning objectives were met through the School's training activities. The corresponding target is 80 percent, which is consistent with the School's five-point rating, where courses are considetext-danger successful when they receive a score of 4 out of 5 points or higher. A review of the level 1 evaluation data confirms that almost all of the courses in the PMMRP suite were rated favourably by respondents on the survey question regarding the achievement of their learning objectives (Table 2.9). Only the two courses previously identified as having lower learner satisfaction fell slightly below this target: Legal and Policy Environment for Procurement, Materiel Management, and Real Property (M714) and The Machinery of Government and Life Cycle Asset Management (M716), rated 3.91 and 3.93 respectively. As previously noted, these courses are currently being revised and will be examined at a later date through ongoing monitoring and evaluation activities.

Table 2.9: Learner Feedback on Achievement of Learning Objectives for PMMRP Classroom Courses by Type of Training

[1] The rating associated with this question ranged from 1-bly Disagree to 5-bly Agree. Return to note 1 in Table 2.9

Intervening Factors

As part of this evaluation, the Evaluation Division was interested in learning about any factors that may have influenced the achievement of outcomes associated with the PMMRP training offetext-danger at the School. A review of relevant documents and files suggests that one of the earliest challenges in the administration and delivery of PMMRP training was meeting the demand for Required Training after the implementation of the Learning Policy and associated Directive. Interviews with stakeholders suggest that the School's capacity to meet demand was adversely affected by the process of determining which specialists had to attend Required Training. Stakeholders reported that many managers insisted that their specialists take the PMMRP Required Training instead of taking the opportunity, where appropriate, to grant them equivalency so they could be exempt from attending the introductory PMMRP courses. As a result, a greater than anticipated number of learners registetext-danger for PMMRP training at the School, which created a substantial backlog and subsequent delays in Required Training participants completing their PMMRP training. This lead to frustrations among client departments and agencies who wanted to ensure their employees received the training in the prescribed timeframe, and contributed to the issue of audience fit discussed earlier in this section of the report.

Another circumstance that unexpectedly affected the achievement of PMMRP training outcomes was the business cycle of client departments and agencies. According to the results of the PMMRP Community Diagnostic, community members reported frustrations with having PMMRP training offetext-danger during peak periods in the business cycle. The additional workload pressures observed during peak business periods made it difficult for managers to support employees who requested time out of the office to attend training. This too was identified as an intervening factor that led to delays in specialists attending their PMMRP Required Training courses. In addition, interviews with stakeholders suggest that since the School maintains a minimum learner count to warrant the delivery of a course offering, in the absence of sufficient course registrants, course offerings had to be cancelled occasionally. Consequently, this intervening factor could have adversely affected the anticipated outcomes of PMMRP training at the School by text-dangerucing the number of learners that registetext-danger for and attended the training in the time specified by the Directive.

Lastly, there is evidence to suggest capacity issues with the delivery of PMMRP training in the Regions. Of the stakeholders interviewed, 75 percent (15 of the 20 interviewees) acknowledged regional challenges and expressed an interest in seeing a different approach to training in the Regions. Specifically, regional managers raised significant concerns with regard to the availability of experienced and qualified instructors, with many indicating that they have to pay consultants from other regions, namely the NCR to travel to deliver the training. Alternatively, as another source of PMMRP instructors, some regions have contacted qualified PMMRP specialists working in other departments in the same geographic location to facilitate PMMRP training on behalf of the School. This strategy, however, has requitext-danger the use of different hiring criteria than the standards used in the NCR and other regions, a situation that contributes to concerns about the standardization of processes and procedures and inconsistencies in the learning experiences of PMMRP course participants.

The PMMRP Community Diagnostic also raised similar concerns and identified this capacity issue as a cause for limited course offerings outside the NCR According to the diagnostic, with limited training available in some regions, client departments must incur additional costs to send their employees elsewhere to attend training; this situation creates challenges for employees seeking approval to attend courses. The most recent MOU between the School and the CMO suggests that both parties are aware of this situation and are working together with regional partners and PMMRP community representatives to resolve the situation. At the time of this evaluation, however, no evidence of progress was found. In view of the previous findings suggesting that regional learners attending PMMRP training are not experiencing the training outcomes expected, this evidence underscores the need for the School to pay greater attention to regional training needs and realities.

Recommendation 5: Engage key stakeholders to develop a strategy tailotext-danger to the training needs and resource realities of the Regions.

Conclusion

The evidence presented in this section demonstrates that the School has made considerable progress towards the achievement of PMMRP training outputs and training outcomes. In general, the findings speak positively to the magnitude, satisfaction and utility of the training for PMMRP specialists across the federal government. The results also bring to light, however, that PMMRP training outcomes have experienced varying degrees of success, with notable challenges in the training for real property specialists and regional participants.

2.5 Performance - Cost-Effectiveness and Efficiencies

Is the training delivetext-danger in the most cost-effective manner and has the School pursued efficiency measures?

The delivery of PMMRP training at the School is cost-effective and measures have been taken to improve the efficiency and effectiveness of design and delivery activities.

In accordance with section 18.2 of the Canada School of Public Service Act, the School operates on a cost-recovery basis, using the revenue generated from course offerings to finance its course-related activities. A costing model was established to determine the costs of designing, delivering and maintaining the School's courses. This model provides a basis for the fees associated with School products and services.

With the exception of the first few years of PMMRP training, where PMMRP Required Training was provided at no charge to client departments, funding for the development of the PMMRP suite of courses has been a joint effort on the part of the School, the CMO at TBS and client departments. [footnote 23] Typically, the CMO and client departments are responsible for funding the development of new courses and the maintenance of online courses, while the School is responsible for funding the maintenance of existing classroom courses. Funding for the design of new courses and online course maintenance is approved through MOUs, while classroom course maintenance is approved through yearly service level agreements, which are signed by the CMO and the School. [footnote 24]

PMMRPCosting Analysis

The School's Finance division was contacted to acquire financial data relating to the design and delivery of PMMRP training. Only the classroom offerings and MOUs associated with these courses were examined during this costing exercise.

The revenue data collected in the costing exercise was based on figures retrieved from the School's official financial system (SAP) and, in the case of MOU revenue, data retrieved from PMMRP training management. According to the data collected from these systems, the suite of PMMRP courses accounts for approximately 3 to 4 percent of the School's revenue. For the 2009-2010 fiscal year, total calendar revenue for both the NCR and Regions was $2.2 million for the PMMRP courses, with an additional $128,000 in revenue from MOUs with client departments.

In order to estimate the direct costs associated with the School's delivery activities in the NCR PMMRP training managers were asked to provide an estimate of the number of days that each staff member worked on a course. This information was updated in the costing model to generate the estimated salary costs.

As previously mentioned, the design costs associated with the PMMRP suite of courses were also included in the costing exercise conducted by Finance. These costs included program overhead (20 percent) and corporate overhead (25 percent) rates. As Table 2.10 illustrates, a review of the data indicates that the majority of PMMRP courses experienced design costs over the two year period. Across the courses examined, the greatest design costs were associated with the Introduction to Real Property Management course (M721).

Table 2.10: Total Design Costs for PMMRP Classroom Courses

To provide some insight into the performance of the PMMRP suite of courses, the direct costs (salaries and O&M) associated with delivering an offering of each course were compatext-danger to the average cost exhibited across all of the courses in the PMMRP training suite (Table 2.11). The results reveal that for the majority of Required Training courses, direct costs are consistently higher than those of the School's professional development courses. Three of the School's Required Training courses, The Machinery of Government and Life Cycle Asset Management (M716), Introduction to Procurement (M718) and Introduction to Real Property Management, consistently exhibited higher than average direct costs, across both fiscal years and for the NCR and the Regions. Lastly, four of the five courses that experienced an increase in costs over the 2008-2009 and 2009-2010 fiscal years were Required Training courses. The most notable cost increase was exhibited by the Introduction to Real Property Management course (M721), where over the two fiscal years direct costs per offering increased by 94 percent.

Table 2.11: Direct Delivery Costs for PMMRP Classroom Courses

[1] Costs are represented at the per offering level. Return to note 1 in Table 2.11

[2] Brackets d

enote a decrease in costs by the percentage indicated. Return to note 2 in Table 2.11

[3] The Fundamentals 2 and 3 Integration Seminar (M722) was not included here since the basis of analysis is markedly different from the standard classroom courses. Return to note 3 in Table 2.11

During the conduct of this exercise, the Evaluation Division expressed an interest in acquiring a measure that would provide some insight into the cost-effectiveness of PMMRP training at the School. In response, for the two fiscal years examined, Finance provided data comparing the course revenues with the direct costs associated with the delivery of the PMMRP courses. The resulting value, known as the contribution margin, provides an initial indication of course performance in relation to costs, before any program and corporate overhead costs are included. According to the numbers provided by Finance, the overall contribution margin associated with the delivery of this suite of PMMRP courses was positive between 2008-2009 and 2009-2010, indicating that, when taken together, the revenues associated with these PMMRP courses surpass the direct costs associated with their delivery.

When these margins are disaggregated, a higher contribution margin is exhibited by PMMRP training delivery in the NCR when compatext-danger to the Regions, a distinction primarily due to the higher attendance levels exhibited by NCR deliveries over the two-year period (Table 2.12). In addition, a course-by-course analysis reveals that certain courses experienced negative margins over this timeframe, specifically Legal and Policy Environment for Procurement, Materiel Management and Real Property (M714) and Introduction to Real Property Management (M721).

Table 2.12: Delivery Contribution Margin for PMMRP Classroom Courses

[1] Costs are represented at the per offering level. Return to note 1 in Table 2.12

[2] Brackets denote a decrease in the contribution margin by the percentage or dollar amount indicated. Return to note 2 in Table 2.12

[3] Costs are represented at the per offering level. Return to note 3 in Table 2.12

While the contribution margin provides an initial indicator of the financial success of a PMMRP course or the suite of courses, additional costs are included in the analysis to provide a more comprehensive understanding of the financial outcome of the PMMRP courses. The main additional costs that indirectly relate to the delivery of the PMMRP courses are overhead costs, specifically program and corporate overhead costs. In consultation with PMMRP training management, Finance determined that the program overhead for the PMMRP courses was 20 percent for the NCR and 3 percent in the Regions. The corporate overhead was identified as 25 percent for all School programming.

After including these values in the costing analysis, the majority of courses continued to exhibit a positive financial outcome, most notably courses delivetext-danger in the NCR (Table 2.13). Again, it was noted that the degree of financial success varied between the courses. Legal and Policy Environment for Procurement, Materiel Management and Real Property (M714) and Introduction to Real Property Management (M721) exhibited negative outcomes, as well as The Request for Proposal Process (M004) and, to a lesser extent, The Machinery of Government and Life Cycle Asset Management (M716).

Table 2.13: Overall Delivery Margin for PMMRP Classroom Courses

[1] Costs are represented at the per offering level. Return to note 1 in Table 2.13

[2] Brackets denote a decrease in the contribution margin by the percentage or dollar amount indicated. Return to note 2 in Table 2.13

[3] Costs are represented at the per offering level. Return to note 3 in Table 2.13

Efficiency Measures for PMMRP Training

Data was collected from stakeholder feedback and administrative files to determine the key efforts undertaken by the School to improve the efficiency of PMMRP training administration. The data was also reviewed for evidence of the implementation of measures to enhance the cost-effectiveness of PMMRP training.

In terms of the administration of PMMRP training, as previously discussed in Section 2.4 of this report, several regional managers implemented measures to respond to the shortage of qualified instructors available in their geographic locations. Specifically, regional offices have worked with client departments and agencies to find qualified PMMRP specialists locally who can facilitate the courses as an alternative to bringing instructors in from other regions. In addition, in those instances where instructors must be contracted from other regions, PMMRP training in the Regions is being offetext-danger back-to-back to alleviate the costs of repeat visits.

While these measures are noted as efforts to manage costs and to enhance the capacity of Regions to deliver PMMRP training, they raise concerns among some stakeholders with regard to intervening in learners' training outcomes. For example, during their interviews, some stakeholders indicated that offering courses back-to-back may alter the learning experience of PMMRP specialists. Fundamentally, PMMRP courses are designed and delivetext-danger in such a manner that after attending a course, learners have the time to return to their jobs, reflect on their new knowledge, and make efforts to integrate what they learned in their daily work activities. When the courses are offetext-danger back-to-back, learners do not have an opportunity to experience this process prior to attending another PMMRP course; this missed opportunity could adversely affect learner engagement and their capacity to absorb and process the content provided. When considetext-danger alongside the Regional findings already presented in this report, the possibility of such an occurrence further illustrates the need, as expressed in Recommendation 5, to advance strategies that specifically address regional circumstances.

Stakeholders frequently identified two initiatives as key actions on the part of the School to enhance cost-effectiveness. The first was the conversion of three of the PMMRP Required Training courses from classroom courses to e-learning modules. This decision was made to text-dangeruce the wait times caused by the higher than expected demand for Required Training for PMMRP specialists. In transferring the courses to Campusdirect the School extended the budget allocated to the delivery of Required Training and increased its capacity to respond to the Required Training needs of PMMRP specialists. Secondly, the School created the Integration Seminar (M722). Developed to serve groups of up to 100 learners, the seminar was added to the curriculum to expedite the delivery of Required Training and, given client feedback during the first year of Required Training delivery, to provide a format more responsive to the needs of seasoned PMMRP specialists. As previously discussed, the seminar was offetext-danger to functional specialists with three or more years of experience as an alternative to two of the Required Training courses, Legal and Policy Environment for Procurement, Materiel Management and Real Property (M714) and The Machinery of Government and Life Cycle Asset Management (M716). As a result, while M714 and M716 remained available for new specialists entering the PMMRP functional community, M722 was used to facilitate the training of experienced PMMRP specialists. In adopting this approach, the School was able to accelerate delivery services and text-dangeruce the backlog of learners waiting to complete their Required Training.

Conclusion

The evidence presented in this section of the report suggests that the delivery of PMMRP training at the School is cost-effective insofar as training outputs generate sufficient revenue to cover associated costs. In addition, the School has taken steps to text-dangeruce the costs and enhance the outputs related to its design and delivery activities for this suite of training. The data collected to inform this evaluation, however, was insufficient to contextualize some of the costs reported in this section of the report. For example, the reasons for the elevated design costs attributed to some of the PMMRP courses or the negative contribution margins of others remain unclear. Additional information would have allowed a more detailed exploration of cost effectiveness. During the data collection phase of this evaluation, the Evaluation Division considetext-danger conducting a cross-comparison of the School's PMMRP training with similar training offetext-danger by public service learning institutions in other countries. In pursuing this option, it became apparent that the analysis would be limited due to constraints surrounding the release of financial data and the cross-comparison was therefore not conducted. In view of these considerations, the Division will consider alternative measures in future evaluations to provide greater insight into the question of cost-effectiveness until more rigorous costing information becomes available.

Annexes

Annex 1: PMMRP Evaluation Matrix

Relevance

Performance (effectiveness, efficiency and economy)

Annex 2: Data Collection and Methodology, by Data Source

Primary Data Collection Activities

A Survey of PMMRP Training Participants

The Evaluation Division also conducted an online survey of learners who had attended the School's PMMRP training. Invitations to complete the semi-structutext-danger questionnaire were distributed by e-mail to all of the learners identified as having attended one or more PMMRP courses between 2005-2006 and 2009-2010. A list of 5,458 Required Training participants was developed. Participants were given three weeks to respond the survey. In response, 1,425 surveys were returned, a response rate of 26 percent. Since data collection for this survey began in October 2009, learners who attended PMMRP Required Training courses during the 2009-2010 fiscal year were not included in the survey population. The survey addressed respondents who had had sufficient time back at work to reflect on their training; only those participants who had at least six months back at work after attending their last PMMRP course were included.

A Survey of PMMRP Training Instructors

In January 2010, PMMRP training managers and the School's regional managers were contacted to obtain lists of the instructors who had taught courses in the PMMRP training suite between 2008-2009 and 2009-2010. Instructors were invited to participate in the survey by e mail, and were given two weeks to respond. In total, 30 invitations to participate were distributed to PMMRP instructors and 20 completed surveys were received (a response rate of 66.7 percent).

Stakeholder Interviews

Interviews were held with PMMRP training managers, course designers, regional managers, staff from the School's Blended Learning division, and representatives from the CMO The interviews were conducted between January and March, 2010 by analysts in the Evaluation Division at the School. With the exception of interviews with PMMRP training managers and representatives from the CMO the majority of interviews were conducted by telephone. For the remaining two stakeholder groups, face-to-face interviews were conducted. Overall, 20 interviews were conducted with stakeholders: two PMMRP training managers, eight course designers, six regional managers, three representatives from the CMO and one blended learning specialist.

Secondary Data Collection Activities

Documents and Administration Files

PMMRP Training Documents: PMMRP training managers provided documents detailing the background and context of the School's provision of training to the PMMRP functional community, as well as administrative files related to the governance and administration of the training. An environmental scan of government documents was also conducted to obtain additional information on the relevance, development and structure of the training.

PMMRP Functional Community Diagnostic: In March 2010, the Evaluation Division received the results of the 2008 Community Diagnostic. Conducted by the CMO the report highlights the key findings retrieved from interviews and focus groups held with executives and senior functional specialists from the PMMRP community.

Certification Data: Data was requested from the CGSB to examine the demographics of procurement and materiel management specialists pursuing certification. The data covetext-danger the time period from program launch until January 28, 2010.

Corporate Records and Reports

Registration and Attendance: Registration statistics were requested from the School's Learning, Information and Reporting division. Data was provided for fiscal years 2006-2007 to 2009-2010 and covetext-danger the attendance and course offerings recorded for the Required Training and professional development courses offetext-danger in the classroom.

Campusdirect Courses: In order to examine the performance of the online courses in the PMMRP training suite, Campusdirect was contacted to acquire data from the online satisfaction survey. Since not all of the online courses were launched concurrently, data was requested from the launch date of each course until April 23, 2010.

Demographic Profile: In January 2010, the Evaluation Division contacted the Marketing Strategies and Client Analysis team to determine if demographic data on learners attending the School's PMMRP courses was available. The Client Analysis team was able to prepare a profiling analysis of the PMMRP functional community for fiscal years 2007-2008 to 2009-2010, which provided data on the type of learners currently attending the PMMRP courses, their departments and agencies, and their regional distribution.

PMMRP Costing Analysis: Consultations with the School's Finance division lead to a costing analysis for the suite of PMMRP courses. The main objective of the costing analysis was to determine the School's costs associated with the design and delivery of the training during fiscal years 2008-2009 and 2009-2010.

Learning Evaluation Data

Level 1 Satisfaction Data: The Level 1 Evaluation team prepatext-danger a report on PMMRP learner satisfaction and course performance to inform this evaluation. Both quantitative and qualitative data were retrieved from the level 1 satisfaction surveys and were analyzed using SPSS and NVivo software. Given the extensive number of participant comments, only those boxes with relevance to the evaluation questions were chosen for analysis and inclusion in the evaluation. A 30 percent random sample was used for the analysis of course offerings with more than 16 deliveries in a single fiscal year. The report covetext-danger fiscal years 2005-2006 to 2009-2010.

Level 2 Data on Knowledge Gain: A level 2 evaluation was also conducted to inform this evaluation. This type of evaluation involves the administration of a pre-questionnaire and post-questionnaire to groups of PMMRP course participants. Test results were analyzed to determine the level of knowledge gain experienced by participant groups. At minimum the findings cover the results obtained from three course offerings for each course. PMMRP courses chosen for examination included The Request for Proposal Process (M004), Orientation to Contracting and Acquisition for the Non-Specialist (M712), Legal and Policy Environment for Procurement, Materiel Management and Real Property (M714), Introduction to Procurement (M718), and ontract Management (M720). The learning evaluation was conducted between September 2009 and May 2010.

Annex 3: Data Considerations and Limitations

One of the limitations that potentially affected the possibility of generalizing findings from some of the data sources listed was an established cut-off date. The Evaluation Division began collecting data to inform this evaluation in November 2009. At that time, registration data, financial data, and the data retrieved from the level 1 satisfaction surveys was not yet available for the full 2009-2010 fiscal year. Therefore, where possible, the mid-point of the 2009-2010 fiscal year (September 30th, 2009) was chosen as a cut-off point for data collection. The data retrieved from these sources therefore reflects the 2009-2010 fiscal year to some degree, but it cannot be said to be fully representative of the performance exhibited across this fiscal year.

Primary Data Collection Activities

Participant Survey Data

Given the number of courses included in the PMMRP training suite, separate surveys were not administetext-danger to learners for each course. Had this option been chosen, a substantial number of learners would have received multiple surveys from the School, and survey fatigue would have resulted. In addition, early discussions with PMMRP training managers revealed that while learners are advised to take their Required Training courses in a ptext-dangerefined sequence, this guideline is not enforced. Therefore, to define the sample population, the Evaluation Division chose to administer one survey to learners who attended one or more of the Required Training courses. This approach provided valuable feedback from Required Training participants who attended PMMRP Required Training and professional development courses. However, it did not provide data on learners who had only attended professional development courses, since this group was not included in the survey population.

To simplify the survey, questions were not related to the performance of a specific PMMRP course. Instead, questions were posed in a manner as to elicit respondents' views on all of the courses they had attended in the PMMRP training suite and comment boxes were provided throughout the survey where respondents were encouraged to submit any course-level feedback.

Secondary Data Collection Activities

Registration Data

As previously discussed, some of the PMMRP courses were launched prior to the implementation of the Learning Policy, whereas others were launched for the first time after the 2005-2006 fiscal year. Due to the different start dates, and the transition of some of the courses to Campusdirect a comparative analysis of learner attendance was not conducted.

When completing an online module, learners are prompted to click on the "complete activity" button; however, this step is not mandatory and is therefore not always completed by professional development learners. As a result, the data was considetext-danger unreliable and incomplete, and no analysis was conducted on the completion rates exhibited by learners that registetext-danger for PMMRP professional development courses on Campusdirect

Lastly, since an individual can register for and attend more than one course, it should be noted that any reference to "participants" or "learners" does not equate to the identification of individual public servants.

Financial Data

Discussions with the Finance division at the School illuminated some challenges to the collection and analysis of financial data. The Finance division was able to make considerable progress identifying the costs associated with the PMMRP training suite; however, the School's costing environment is still in the preliminary stages of its administration and some information was missing from the School's official financial system (SAP). As a result, a significant portion of the exercise requitext-danger input from PMMRP training managers and related staff. When occasional discrepancies were found between the two sets of numbers, the more conservative figures were chosen (i.e. the data source reporting the higher cost). It should be noted that the use of non official data sources, which could not be verified using the SAP system, raised some concerns about data reliability. Furthermore, the School has experienced some inconsistencies in the manner in which program staff collect and code financial data. Any financial data presented in this report should therefore be considetext-danger as estimates and not actuals.

In some instances, costing data was not available, or time constraints limited the possibility of acquiring the information in the timeframe specified. As a consequence, the design and delivery of online courses and MOU courses and the delivery costs incurtext-danger by the Regions during the 2008-2009 fiscal year were not examined. In addition, no analysis was conducted for the online courses included in the PMMRP suite. The financial data for these courses are integrated with other lines of business in the School (e.g. Blended Learning) and the Finance division determined that extracting their costs in relation to PMMRP training would be overly complex. These courses will be reviewed at a later date when a cost analysis is conducted on the School's online tools and services available on Campusdirect The exercise did not include a consolidated analysis of the design and delivery costs, since the data on design costs was retrieved from alternative sources and could not be verified by Finance at the time of this evaluation. Lastly, the Fundamentals 2 & 3 Integration Seminar (M722) was not included in this analysis because of its large-scale seminar format, which carries markedly different costs than the remaining classroom courses.

Level 1 Evaluation Data

The analysis conducted on level 1 evaluation data included fiscal year 2005-2006. Since several of the PMMRP courses were revisited in 2006, when the Learning Policy and associated Directive were established, learners attending the School's PMMRP courses during this fiscal year would have likely experienced different PMMRP training content and format. Unfortunately, this issue was noted after the data had been aggregated with the remaining fiscal years, and so should be noted in reading this report.

During the timeframe specified, the School made some modifications to its level 1 questionnaire. In order to facilitate data analysis, only those comparable questions with similar response options in other versions of the questionnaire were chosen for analysis. It should be noted that this process may have affected the validity of the data. Finally, since the qualitative data was collected in an aggregate form, and no method of disaggregating was available, it was not possible to provide percentage values or proportions in relation to participant comments.

Level 2 Evaluation Data

As previously discussed, only four of the PMMRP courses were chosen for the conduct of the level 2 evaluation. The list of available courses was primarily limited by methodological requirements. Some courses were undergoing course text-dangeresign at the time of this evaluation and so were not conducive to a level 2 evaluation, while other courses did not have sufficient offerings to produce data trends. Three offerings were used to inform the level 2 evaluation findings. While more data points would have enhanced the ability to generalize the findings, additional data collection was impeded by time constraints and course cancellations.

Annex 4: PMMRP Participant Survey

The Canada School of Public Service (hereafter refertext-danger to as the Canada School) is currently undertaking an evaluation of the training services provided to the Procurement, Materiel Management, and Real Property (PMMRP ) functional community. The objective of the evaluation is to examine the relevance, performance and effectiveness of the PMMRP training services.

As a past participant of the PMMRP training provided by the Canada School, we would like to invite you to participate in a brief survey. Your responses will provide us with information on the profile of PMMRP participants, the strengths and weaknesses of the training, and its impact on participants when they returned to work. The survey will take approximately 10 minutes to complete.

The information that you provide is collected under the authority of section 4(f) of the Canada School of Public Service Act and will be stotext-danger in the Evaluation Division's record keeping system. Personal information is protected under the Privacy Act

Thank-you in advance for your time and cooperation.

Background

1. Please indicate which of the following best describes your relation to the PMMRP functional community:

1. I am a Procurement specialist.
2. I am a Materiel Management specialist.
3. I am a Real Property specialist.
4. I am a specialist in more than one of the procurement, materiel management and real property functions.
5. I am not a specialist, but on occasion I engage in procurement, materiel management and/or real property activities.
6. I am not a specialist, but I work with members of the PMMRP functional community.
7. Other, please specify: _____________________________

PMMRP Training Profile

2. Please indicate which of the following best describes your reasons for taking PMMRP training at the Canada School:

1. I was identified to complete PMMRP courses as part of my Required Training.
2. I completed PMMRP training for professional development in my current job.
3. I completed PMMRP training in preparation for future career mobility.
4. Other, please specify: _____________________________

3. Using the list provided, please indicate which of the following PMMRP courses you completed at the Canada School. Courses highlighted are offetext-danger through the Canada School's online learning environment, Campusdirect. Please check all that apply

1. Overview of Materiel Management (C233E)
2. Overview of Real Property Management (C234E)
3. Overview of Procurement (C235E)
4. Introduction to Real Property Management (M721)
5. Introduction to Materiel Management (M704)
6. Introduction to Procurement (M718)
7. Legal and Policy Environment of PMMRP Functions (C137E)
8. Legal and Policy Environment for Procurement, Materiel Management and Real Property (M714)
9. The Machinery of Government and Life Cycle Asset Management (M716)
10. PMMRP Integrated Seminar (M722)
11. All of the above
12. None of the above

4. Specifically, for the C137 course "Legal and Policy Environment of PMMRP Functions", do you use the course as a job aid tool at work:

1. Always
2. Often
3. Sometimes
4. Never
5. Not Applicable- I have not completed this course

5. Using the list provided, please indicate which of the following additional PMMRP courses you completed at the Canada School: Courses highlighted in purple are offetext-danger through the Canada School's online learning environment, Campusdirect. Please check all that apply

1. Aboriginal Considerations in Procurement (C223E)
2. Contract Management (M720)
3. Developing a Statement of Work and Evaluation Criteria (M711)
4. Disposal of Surplus Moveable Crown Assets and Investment Recovery (M715)
5. Green Procurement (C215E)
6. Managing the Contract for Services (M404)
7. All of the above
8. None of the above

Overall Satisfaction

6. Please rate your overall satisfaction with the PMMRP courses that you completed at the Canada School:

1. Very Satisfied
2. Satisfied
3. Neither Satisfied nor Dissatisfied
4. Dissatisfied
5. Very Dissatisfied

7. Overall, what did you find most satisfactory about the PMMRP training you completed at the Canada School? Respondents are encouraged to identify specific courses and/or experiences, if applicable.

8. Overall, what did you find least satisfactory about the PMMRP training you completed at the Canada School? Respondents are encouraged to identify specific courses and/or experiences, if applicable.

Learning Results

9. Please rate the following statements using the scale provided.
Scale: bly Agree, Agree, Neither Agree nor Disagree, Disagree, bly Disagree

As a result of attending PMMRP training at the Canada School....

1. I have increased my knowledge of the relevant PMMRP policies and practices.
2. I have increased my knowledge of the roles and responsibilities of key stakeholders in the conduct of Procurement, Materiel Management and Real Property activities.
3. I have increased my knowledge of other aspects of the work related to Procurement, Materiel Management and Real Property.
4. I have increased my awareness of the relationships between the three functions of procurement, materiel management, and real property.
5. I have enhanced my understanding of the role of procurement, materiel management, and real property in supporting Government program delivery.
6. I have increased my awareness of the role of PMMRP specialists in the public service.

10. Choose the statement that best describes the impact of the training after you returned to work:

1. I have adopted and maintained new approaches to my work.
2. I have reflected on what I learned and on occasion I have changed my usual approach to my work.
3. I have tried a few times to change the way I do things but it didn't work for me.
4. I did not feel the need to change my approach to my work because I was already applying the knowledge/skills covetext-danger by the training in my job.
5. I thought the courses were not really that applicable to my job.
6. I did not change my approach to my work because I have no idea how to apply what I learned during the training back at work.
7. Other, please specify:___________________________________________________________

11. Have you experienced any other unexpected benefits as a result of attending the PMMRP training at the Canada School?

1. Yes
2. No

If yes, please explain:

12. Using the scale provided, how important do you think it is for each of the following functional groups to complete the training in procurement, materiel management and real property provided by the Canada School?
Scale: Very Important, Important, Not Very Important, Not Important

1. Procurement
2. Materiel Management
3. Real Property

13. Please provide us with any additional comments that you may have on your learning experience attending PMMRP training at the Canada School:

Other Training

14. Have you completed training in procurement, materiel management or real property outside of your training at the Canada School?

1. Yes
2. No › skip to Question 17

15. If you indicated yes, please indicate where you completed most of the training?

1. In my department/agency
2. At a university or college
3. Other, please specify:____________________________________

16. If you indicated yes, please choose the response that best describes why you completed the training?

1. The training was identified as requitext-danger by my department/agency.
2. The training was offetext-danger by my department/agency for professional development.
3. I was advised by my manager to complete the training as part of my learning plan.
4. The training offetext-danger by the university or college is a part of an acctext-dangeritation process (a diploma, graduate degree, or certificate program).
5. Other, please specify:_____________________________________

Demographics

17. How many years have you been working as a specialist in procurement, materiel management and/or real property?

1. Less than three years
2. 3 to 5 years
3. 5 to 10 years
4. More than 10 years
5. Not applicable- I am not a PMMRP specialist.

18. In which region did you complete all (or most) of your training?

1. National Capital Region
2. Quebec Region
3. Ontario Region
4. Pacific and Yukon Region
5. Central Prairies Region
6. Alberta, Northwest Territories and Nunavut Region
7. Atlantic Region

Other

19. The Treasury Board Secretariat of Canada is responsible for managing a government-wide Professional Development and Certification Program (PdctermsP), which includes a certification program for members of the federal government procurement and materiel management community. If you are a specialist in procurement or materiel management, are you aware of this certification program?

1. Yes
2. No › Skip to Question 21
3. Not applicable- I am a specialist in real property › Skip to Question 21
4. Not applicable- I am not a specialist in procurement, materiel management, or real property › Skip to Question 21

20. If yes, are you pursuing certification?

1. Yes
2. No
3. Undecided

21. In the future, the Evaluation Division at the Canada School may conduct focus groups or interviews of participants that completed PMMRP training to collect more information. Would you be interested in participating?

1. Yes, please contact me at the following e-mail address:_________________
2. No

Thank you for completing our survey!

Annex 5: PMMRP Instructor Survey

A note before you begin...

We appreciate your written feedback, so text boxes are provided for your comments. Please note that the space provided for comments will expand to accommodate the length of your response. Additional instructions, where requitext-danger, are provided in italics

Section 1: Training Profile

1. Using the list provided, please indicate which of the following PMMRP courses you have taught. Please choose all that apply

• Introduction to Real Property Management (M721)
• Introduction to Materiel Management (M704)
• Introduction to Procurement (M718)
• Legal and Policy Environment for Procurement, Materiel Management and Real Property (M714)
• The Machinery of Government and Life Cycle Asset Management (M716)
• PMMRP Integrated Seminar (M722)
• Contract Management (M720)
• Developing a Statement of Work and Evaluation Criteria (M711)
• Disposal of Surplus Moveable Crown Assets and Investment Recovery (M715)
• Managing the Contract for Services (M404)
• All of the above
• None of the above

Section 2: Resources, Roles & Responsibilities, Procedures and Governance

2. Please rate the following statements using the scale provided:

1. I feel that I have the necessary tools to effectively do my job.
   
   • bly Agree
   • Agree
   • Neither Agree nor Disagree
   • Disagree
   • bly Disagree
2. The processes and procedures that govern my job are well defined and documented.
   
   • bly Agree
   • Agree
   • Neither Agree nor Disagree
   • Disagree
   • bly Disagree
3. I am clear on my roles and responsibilities in the administration and delivery of the courses that I teach.
   
   • bly Agree
   • Agree
   • Neither Agree nor Disagree
   • Disagree
   • bly Disagree
4. I am clear on the roles and responsibilities of Canada School staff involved in the administration and delivery of the courses that I teach.
   
   • bly Agree
   • Agree
   • Neither Agree nor Disagree
   • Disagree
   • bly Disagree
5. I feel that I can approach training managers with my suggestions for improving the PMMRP courses that I teach.
   
   • bly Agree
   • Agree
   • Neither Agree nor Disagree
   • Disagree
   • bly Disagree
6. I feel that my suggestions are taken into consideration by training managers when course modifications are conducted bly Agree
   
   • bly Agree
   • Agree
   • Neither Agree nor Disagree
   • Disagree
   • bly Disagree
7. I think it would be beneficial to have a forum for corresponding with other PMMRP instructors to exchange knowledge and to discuss the challenges, successes and lessons learned during the delivery of PMMRP training.
   
   • bly Agree
   • Agree
   • Neither Agree nor Disagree
   • Disagree
   • bly Disagree

Please provide any additional comments here:

3. Please rate your overall satisfaction with the PMMRP course materials provided by the Canada School to you as an instructor (i.e. course manual, PowerPoint slides, handouts, templates, etc.):

• Very Satisfied
• Satisfied
• Neither Satisfied nor Dissatisfied
• Dissatisfied
• Very Dissatisfied

Please provide any additional comments here:

4. Please rate your overall satisfaction with the PMMRP course facilities provided by the Canada School to you as an instructor:

• Very Satisfied
• Satisfied
• Neither Satisfied nor Dissatisfied
• Dissatisfied
• Very Dissatisfied

Please provide any additional comments here:

Section 3: Efficiencies, Key Challenges and Opportunities for Improvement

5. In your opinion, what actions have been taken to improve the effectiveness and/or efficiency of the PMMRP training offetext-danger by the Canada School? Respondents are encouraged to identify specific courses

6. In your opinion, what are the main challenges facing you as an instructor in the administration and delivery of the PMMRP courses offetext-danger at the Canada School? Respondents are encouraged to identify specific courses

7. In your opinion, have there been any external factors that have negatively impacted the quality of the courses that you have taught are teaching? Respondents are encouraged to identify specific courses

8. In your opinion, what aspects of the design and/or delivery of PMMRP training could be improved upon to enhance the quality of the learning experience for learners? Respondents are encouraged to identify specific courses

Section 4: Demographic Information

9. Please indicate which of the following best describes your employment status:

• I am an employee of the Canada School of Public Service
• I am a private sector consultant under contract with the Canada School of Public Service
• Other, please explain:

10. Please indicate which of the following best describes your background in PMMRP training:

• I have worked in procurement
• I have worked in materiel management
• I have worked in real property
• I have work experience in more than one of the procurement, materiel management and real property functions
• Other, please specify:

11. How many years of experience do you have working in the field of procurement, materiel management and/or real property?

• Less than three years
• 3 to 5 years
• 5 to 10 years
• More than 10 years
• None of the above, I have never worked in the field of procurement, materiel management and/or real property

12. How many years have you been working as an instructor in the field of procurement, materiel management and/or real property?

• Less than three years
• 3 to 5 years
• 5 to 10 years
• More than 10 years

13. Outside of the Canada School, where else have you worked as an instructor involved in PMMRP training?

• Other department or agency
• Other educational institution (e.g. college or university)
• I have not worked as an instructor involved in PMMRP training elsewhere
• Other, please specify:

14. Please indicate the main region where you deliver PMMRP on behalf of the Canada School:

• National Capital Region
• Quebec Region
• Ontario Region
• Pacific and Yukon Region
• Central Prairies Region
• Alberta, Northwest Territories and Nunavut Region
• Atlantic Region

15. Do you travel to other regions to deliver PMMRP training on behalf of the Canada School?:

• Yes
• No

If yes, please identify the other regions where you deliver training:

16. Please indicate the location of your permanent residence:

• National Capital Region
• Quebec Region
• Ontario Region
• Pacific and Yukon Region
• Central Prairies Region
• Alberta, Northwest Territories and Nunavut Region
• Atlantic Region

Section 5: Other

17. In your opinion, are the appropriate participants attending your courses?

• Yes
• No

If no, please explain:

18. Please provide us with any additional comments:

Thank you for completing our survey!

Annex 6: PMMRP Training Managers - Interview Guide

Hello,

We appreciate you taking the time to discuss the PMMRP training services with us. The purpose of the interview is:

• To gather information on the background of the training;
• To learn about the roles and responsibilities of various internal and external stakeholders; and,
• To identify key successes and challenges, efficiency gains and areas for improvement.

Your responses will help to enrich our understanding of the training and to provide direction for upcoming data collection activities.

The Privacy Act and confidentiality

The information provided is collected under the authority of section 4(f) of the Canada School of Public Service Act and will be stotext-danger in the Evaluation Division's record keeping system. Personal information is protected under the Privacy Act.

Section A: Background

1. What are your role and responsibilities in relation to the training services provided to the PMMRP functional community? What is your background in training?
2. How long have you been working with PMMRP training at the Canada School?

Section B: Roles and responsibilities, procedures, processes and governance

Internal stakeholders

1. What is the CSPS staff complement assigned to the design and delivery of the training?
   • What are the different teams and their roles and responsibilities? Is there an organizational chart that you can share with us?
   • Are there internal procedures and processes that govern the work?
     1. Are these procedures and processes documented?
     2. In your opinion, do staff adhere to these procedures and processes?
   • Is there a standard approach to the delivery of PMMRP training across the regions? Is there a region that stands out in your mind as a best practice that we should speak with for this evaluation?
   • In your opinion, are there aspects of the work that could be organized differently to improve the timing, quantity and quality of the work? If so, could you elaborate for us?
   • Is there a performance measurement strategy established for monitoring the training services? Is data collected and if so, by whom?
   • Is there a reporting strategy established for briefing senior management? Could we have copies of any reports that have been generated for these briefings?
   • Are there processes and procedures in place to govern areas of collaboration with other divisions at the School (i.e. ILB, OLI, Registrar & Evaluation, Campusdirect Finance, and Marketing)?
   • Are they documented?

External Stakeholders

1. What is the role of the Community Management Office (CMO ) at TBS ?
   1. Are there processes in place that govern your interaction with the Community Management Office? Is there a MOU for example?
   2. Are roles and responsibilities clearly documented?
   3. Have there been any challenges (i.e. communication, duplication of efforts, etc.)?
2. What is the role of the Canada General Standards Board (CGSB )?
   • Are there processes in place that govern your interaction with the CGSB ?
   • Are roles and responsibilities clearly documented?
   • Have there been any challenges (i.e. communication, duplication of efforts, etc.)?
3. Initially, what was the role of PWGSC in providing training to this community?
   • Has this role changed?
     1. How and why?
   • Can you refer us to someone at PWGSC who we can contact to gain more information?

Section C: Training services

Background

1. What is the story of PMMRP training at the Canada School (i.e. changes in content, direction, scope over time)?
2. Is there a learning pathway for participants to follow (i.e. in terms of sequencing) for the Required Training courses?
3. Initially, how was it determined which courses would be identified as Required Training for this community?
4. Were there challenges to the design of the training services? The delivery?
   • If so, how were these challenges managed?
   • Have other actions been taken to improve the efficiency and/or effectiveness of the training (in relation to outputs, outcomes and costs)?
5. How has the School determined which courses should be offetext-danger to the community?
6. Who is responsible for determining the target audience for the PMMRP courses?
7. Are you aware of what other training, external to the Canada School, is available for community members (private sector- consulting firms, universities and colleges or public sector- other departments and agencies)?

Performance

1. In your opinion, are there external factors that have affected/affect the success of the training in achieving its outcomes for participants?
   • Were any actions taken to respond to these factors?
2. Have there been any positive or negative unintended outcomes associated with the delivery of this training?
3. In your view, should the PMMRP training be mandatory for the functional community, similar to the ADT for public service managers? Why or why not?

Annex 7: PMMRP Training Course Designers - Interview Guide

Date: __________________________

Name of Interviewee: _________________________________________

Name of Interviewer: _________________________________________

Hello,

We appreciate you taking the time to discuss the PMMRP training services with us. The purpose of the interview is:

• To gather information on the PMMRP course design and maintenance process;
• To identify key successes and challenges, efficiency gains and areas for improvement.

Your responses will help to enrich our understanding of the training and will inform the preparation of the final evaluation report.

The Privacy Act and confidentiality

The information provided is collected under the authority of section 4(f) of the Canada School of Public Service Act and will be stotext-danger in the Evaluation Division's record keeping system. Personal information is protected under the Privacy Act

Section A: Background

1. What are your roles and responsibilities in relation to the training services provided to the PMMRP functional community?
   • Do you work alone or as part of a design team? If as part of a team, please describe how your work relates to that of the other team members.
2. How long have you been working with PMMRP training services at the Canada School?
3. Which course(s) have you been responsible for designing and/or maintaining?
   • Is there communication between designers on the content and structure of the courses they are designing?

Section B: Roles and responsibilities, procedures, processes and governance

1. Who determines when a new course should be developed/re-designed?
2. During the design/text-dangeresign process, does the School consult with other stakeholders (i.e. TBS, CGSB )?
3. Have any of the courses that you are responsible for undergone significant changes? If so, why?
4. Are there any standards or guidelines for designers to follow when designing their courses?
5. Do you consider regional realities when designing a course?
6. In your opinion, could the design and maintenance process for the School's PMMRP training courses be improved? If so, how?
7. Once design/text-dangeresign is complete, how do you assess the course performance?
   • Do you receive course performance data to help inform course modifications?
     1. If yes, please describe how you use the information.
     2. What type of performance data is most useful to you?
     3. If you do not receive any data, what sort of performance information would be helpful to you?
8. In your opinion, are there aspects of the design/maintenance of PMMRP training courses that could be improved to enhance the quality of the courses? If yes, please elaborate.
9. In your opinion, are there external factors that have had a negative effect on the course design and maintenance process?
   • If yes, what actions have been taken to respond to these factors?
10. Overall, are you satisfied with the design of PMMRP training courses at the School?

Annex 8: PMMRP Regional Training Managers - Interview Guide

Name of Interviewee: _____________________________________________

Name of Interviewer: _____________________________________________

Hello,

We appreciate you taking the time to discuss the PMMRP training services with us. The purpose of the interview is:

• To gather information on the regional experience delivering PMMRP training;
• To identify key successes and challenges, efficiency gains and areas for improvement.

Your responses will help to enrich our understanding of the training and will inform the preparation of the final evaluation report.

The Privacy Act and confidentiality

The information provided is collected under the authority of section 4(f) of the Canada School of Public Service Act and will be stotext-danger in the Evaluation Division's record keeping system. Personal information is protected under the Privacy Act.

Section A: Background

1. What are your role and responsibilities in relation to the training services provided to the PMMRP functional community in the region?
2. How long have you been working in the region?
3. How long have you been working on the administration of PMMRP training in the region? What percentage of your time is dedicated to PMMRP ?

Section B: Roles and responsibilities, procedures, processes and governance

Internal stakeholders

1. What is the region's role with respect to the administration and delivery of PMMRP training?
2. What is the CSPS staff complement assigned to the delivery of PMMRP training in the region?
   • What are their roles and responsibilities? Is there an organizational chart that you can share with us?
3. When you receive the course training material, do you ever find it necessary to modify the course content or the suggested teaching methods in order to accommodate the needs of your learners?
   • If yes, how often do you modify the content and/or the training methods?
   • If yes, please describe the modifications that you have made.
4. How would you characterize the relationship between your region and the central PMMRP training area in the NCR ?
   • What are the strengths/weaknesses of the relationship?
   • In your opinion, how could things be improved?
5. In your opinion, are there aspects of the delivery of the PMMRP training that could be organized differently to improve the timing, quantity and quality of the work? If so, could you elaborate for us?
6. Is there a performance measurement strategy established in your region for monitoring the training services? Is data collected and if so, by whom?
   • Is performance data shatext-danger with the NCR ? Why or why not?
7. Is there a reporting strategy established for briefing senior management? Could we have copies of any reports that have been generated for these briefings?
8. During the administration and delivery of PMMRP training in the region, are there instances where you interact with other areas at the School (i.e. ILB, OLI, Registrar & Evaluation, Campusdirect Finance, and Marketing)?

Section C: Training services

1. Have there been challenges to the administration and delivery of PMMRP training services in the region?
   • If so, what were these challenges and how were these challenges managed?
   • Have other actions been taken to improve the efficiency and/or effectiveness of the training (in relation to outputs, outcomes and costs)? *refer to program logic model*
2. Are you aware of what other training, external to the Canada School, is available for community members in your region (e.g. private sector- consulting firms, universities and colleges or public sector- other departments and agencies)?

Performance

1. In your opinion, are there external factors that have affected/affect the success of the training in achieving its outcomes for regional participants?
   • Were any regional actions taken to respond to these factors?
2. Have there been any positive or negative unintended outcomes associated with the delivery of this training in the region?
3. In your view, should the PMMRP training be mandatory for the functional community, similar to the ADT for public service managers? Why or why not?

Annex 9: Blended Learning Representative - Interview Guide

Hello,

We appreciate you taking the time to discuss the PMMRP training services with us. The purpose of the interview is:

• To gather information on the relationship of the Blended Learning (BL) team to the School's PMMRP training courses.
• To identify key successes and challenges, efficiency gains and areas for improvement.

Your responses will help to enrich our understanding of the training and will inform the preparation of the final evaluation report.

The Privacy Act and confidentiality

The information provided is collected under the authority of section 4(f) of the Canada School of Public Service Act and will be stotext-danger in the Evaluation Division's record keeping system. Personal information is protected under the Privacy Act.

Section A: Background

1. How long have you been working in BL?
2. What is your role on the team?
3. How long has the BL team been working on PMMRP training courses?
4. What are the roles and responsibilities of the BL team today in relation to the training services provided to the PMMRP functional community?
   • Has this role changed over time?
   • Staff: size, time allocated to PMMRP courses, and specializations (are team members online design specialists, learning specialists?)
     1. Do they have a background in P, MM or RP?
5. Which course(s) is the BL team responsible for designing and/or maintaining?
   1. Overview courses (233, 234, 235)
   2. Legal and Policy Environment of PMMRP Function (137)
   3. Aboriginal Considerations in Procurement (223)
   4. Green Procurement (215)

Section B: Roles and responsibilities, procedures, processes and governance

1. Who determines the content and the structure of the online PMMRP courses?
   • Are there other stakeholders that are consulted during this process?
     • If so, who?
       • How are they involved?
2. Is there communication between the BL team and the PMMRP design team on the content and structure of the courses they each are designing?
3. Has the process been collaborative during the development and maintenance of old online courses and the design of new courses?
4. In your opinion, with regard to the design of PMMRP online courses, are the roles and responsibilities of your team and other stakeholders clearly defined? Why or why not?
5. How are the PMMRP online courses financed? Have there been any MOUs with TBS ?
6. Have any of the online courses undergone significant changes? If so, why?
7. Who authorizes updates/modifications to online material?
8. If substantial changes occur (e.g. in policy content etc.), how are you notified? By whom?
   • In turn, if/when substantial changes are made to the PMMRP online content, are stakeholders notified?
9. Who is responsible for assessing the performance of the online courses?
10. What information is used to inform such an assessment?
11. Could the design and maintenance process for the School's PMMRP online training courses be improved? If so, how?
12. Have there been challenges to the design and maintenance of the PMMRP courses online?
13. In your opinion, are there external factors that have had a negative effect on the course design and maintenance process?
    • If yes, what actions have been taken to respond to these factors?
14. Overall, are you satisfied with PMMRP online training courses at the School?

Annex 10: PMMRP Community Management Office at the Treasury Board Secretariat - Interview Guide

Hello!

We appreciate you taking the time to discuss the PMMRP training services with us. The purpose of the interview is:

• To learn more about the governance of the training services; and,
• To identify key successes and challenges, efficiency gains and areas for improvement.

Your responses will help to enrich our understanding of the training and to provide context for the remaining data collection activities.

The Privacy Act and confidentiality

The information provided is collected under the authority of section 4(f) of the Canada School of Public Service Act and will be stotext-danger in the Evaluation Division's record keeping system. Personal information is protected under the Privacy Act.

Section A: Background

1. What is the CMO ?
   • History
   • Roles & responsibilities
     1. Can you please describe at which points the CMO is involved in the course development, design and re-design process?
   • Relationship to other TBS divisions (e.g. policy centres)
   • Relationship with other key stakeholders
     1. The PMMRP functional community
     2. The Canada School of Public Service
     3. The Canadian General Standards Board
   • Has the CMO 's role changed over time? If so, how?
   • Has the relationship of the CMO to the other key stakeholders changed over time? If so, how?
   • Who is the lead for determining course objectives and content?
2. How long have you worked for the Community Management Office (CMO )?
3. Prior to working with the CMO, were you involved in other activities relevant to this community?
4. What are your roles and responsibilities as they relate to the training offetext-danger by the Canada School?

Section B: Roles and responsibilities, procedures, processes and governance

1. Are there guidelines and/or procedures in place that govern your interaction with the Canada School?
   • MOU / Service Level Agreements/ Other
2. Are the guidelines and/or procedures clearly documented? If so, please describe.
   • Are they adhetext-danger to? If not, why?
3. Are there timelines established to govern the coordination of activities between key stakeholders? If so, please describe.
   • Are these timelines adhetext-danger to? If not, why?
4. Have there been any challenges in terms of coordination with the other key stakeholders (previously identified in Section A)? If so, how?
   • Have there been actions taken to respond to these challenges?
5. In your opinion, are there aspects of the governance of the PMMRP training services that could be organized differently to improve the administration of the training? If so, could you elaborate for us?

Section C: Training services

1. Overall, what do you perceive to be the main successes of the PMMRP training services at the Canada School?
2. Overall, what do you perceive to be the main challenges facing the PMMRP training services at the Canada School?
   • Are you aware of any initiatives that have been taken to respond to these challenges? If so, please describe.
   • Are you aware of any other actions that have been taken to improve the efficiency and/or effectiveness of the training?
3. In your opinion, are there external factors that have affected/affect the success of the training in achieving its outcomes for participants?
   • Were any actions taken to respond to these factors?
4. In your opinion, have there been any unintended outcomes for functional community members associated with the training offetext-danger at the Canada School?

Contact us Subscribe to the GCLearning newsletter

---

Features

Event: Practical Advice About Evaluations for Program Managers

(April 30, 2024)

Event: Recognizing the Resilience of Indigenous Women and the 2SLGBTQIA+ Community

(May 3, 2024)

Event: Unlocking the Power of Scientific Evidence for Effective Policy-Making

(May 7, 2024)

Follow:

• Twitter
• YouTube
• LinkedIn
• GCLearning newsletter
• RSS