This page has been archived.
Information identified as archived on the Web is for reference, research or recordkeeping purposes. It has not been altered or updated after the date of archiving. Web pages that are archived on the Web are not subject to the Government of Canada Web Standards. As per the Communications Policy of the Government of Canada, you can request alternate formats on the "Contact Us" page.
FINAL
Approved by Secretary
December 6, 2011
This report presents the findings of an evaluation of the Strategic Investment Framework (SIF), which was a fund established to guide investments in implementing the Public Service Modernization Act (PSMA).
The objective of this evaluation was to examine the implementation, relevance and performance (efficiency, effectiveness and economy) of the investments made under the SIF, the extent to which they facilitated achieving outcomes of the PSMA, and the lessons learned from this initiative. The scope of the evaluation focused on the departments that accessed SIF funds, which included central agencies. This evaluation did not assess the effectiveness of the tools or instruments implemented by these departments. Although the SIF was central to the PSMA implementation, this was not an evaluation of the PSMA but of the use of SIF funds.[1]
Between 2004 and 2009 $200 million in SIF funding was provided to 40 departments and agencies to support the achievement of the PSMA's objectives, which were as follows:
The funds were applied to five notional envelopes: institutional change, PSMA human resources (HR) information technology (IT) systems, people preparation, new functions, and project management.
The methodology for the evaluation incorporated the following lines of evidence:
A key feature of this evaluation was that this initiative had ended prior to the evaluation's implementation. As a result, rather than recommendations, lessons learned are presented that could be applied to future funding programs that have a similar design. In addition, the legislative review of the PSMA was undertaken at a later time, during which the environment evolved and, therefore, may have arrived at additional insights and/or conclusions.
A number of limitations, including the timing of the evaluation after the initiative ended, resulted in restrictions on the availability of data to support the evaluation. For instance, the department in which the SIF Project Office was situated was reorganized three times during the life of the initiative, which resulted in the inability to locate many knowledgeable potential informants. This also meant that a survey could not be undertaken.
The SIF appears to have been well implemented, resulting in outputs being created to appropriately guide SIF funding to projects that were consistent with short-term outcomes. There was sufficient evidence for the evaluation to make conclusions on immediate and short-term outcomes but not enough to make conclusions on medium- and long-term outcomes. This should not be interpreted to mean that these outcomes were not achieved, but rather that they could not be measured or assessed based on the available evidence.
SIF-sponsored activities were undertaken throughout departments and agencies to support developing human resources management (HRM) capacity needed to implement the PSMA. The evidence showed that these activities were relevant. In fact, all key informants stated that there remains an ongoing need for the types of activities that were funded by the SIF:
Evaluation questions relating to efficiency focused mainly on how effectively the initiative had been implemented. In this regard, the evaluation results indicate that the SIF was organized and implemented as expected. Ninety-four per cent of SIF funding was disbursed to departments and agencies, and 83.4 per cent of the funds approved were invested by departments and agencies.
Although the Master Plan[2] was issued a year later than the SIF, the first progress report indicated that the PSMA Secretariat developed and maintained project implementation timetables to identify key activities and critical milestones. Status and summary reports were later used to inform the commitments and achievements. Evidence shows that all PSMA legislative requirements were met to a significant degree. There is evidence on the project plan's effectiveness in guiding horizontal initiatives across departments and agencies:
Although results from the document review and from key informant interviews showed that SIF funding did indeed provide value, the evaluation could not determine if this could have been accomplished with fewer resources.
The evidence shows that the level and nature of uptake of SIF funds was appropriate. Eighty-eight per cent of departments and agencies were assisted either directly or indirectly through the Small Agencies Transition Support Team (SATST) or parent departments with their PSMA implementation requirements.
The evaluation could not make a firm conclusion on the extent to which the SIF investment strategy contributed to achieving medium- and long-term outcomes of the PSMA, in part because of a lack of informants and reduced reporting among funding recipients.
Nonetheless, the expected outputs and most immediate and short-term outcomes were achieved. Outputs such as guidelines, communications and training led to the immediate outcomes, thus resulting in departments applying and receiving funding for projects that aligned with PSMA priorities and principles, although the timeliness of implementation was identified as an issue. The short-term outcomes, for the most part, also appear to have been achieved, as evidenced by institutional change[3], HR IT systems, "people preparation" events, and new HR functions to support PSMA implementation. However, the evaluation was unable to draw a conclusion on the extent to which these, in turn, supported public service HR needs, resource processes, cultural change and accountabilities. Similarly, conclusions could not be drawn regarding the long-term outcomes of hiring the right people, collaborative labour–management relations, increased focus on learning and training for employees at all levels, and improved clarity in roles and accountability. These represent a significant limitation of the evaluation.
Many of the lessons learned from this evaluation relate to the issue of performance measurement and reporting. Although the reorganizations of PSHRMAC and the CPSA highlighted the issue of information management, greater awareness is needed regarding the importance of establishing performance measurement strategies and reporting frameworks before implementing government initiatives. Such strategies and frameworks would provide senior management with the information needed on their impact.
This report presents the findings of an evaluation of the Strategic Investment Framework (SIF), which was a fund established to guide investments in implementing the Public Service Modernization Act (PSMA). The evaluation was conducted between December 2009 and March 2010 by Goss Gilroy Inc., and the report was finalized by the Internal Audit and Evaluation Bureau of the Treasury Board of Canada Secretariat.
The legislative review of the PSMA was undertaken after the evaluation of the SIF, during which the environment continued to change. As a result, there may be additional insights or conclusions brought to light with regard to the PSMA itself.
In 2003, when the PSMA Reserve Fund was established, the elements (questions) to be evaluated were identified that formed the basis of this evaluation. These questions focused on implementation rather than results and influenced the collection of data during implementation. Because this evaluation was conducted between 2009 and 2010, it also included the assessment of relevance in order to meet the evaluation standards of the Policy on Evaluation, published in 2009.
The evaluation could not conclude on the achievement of the PSMA-SIF medium- and long-term outcomes mainly because of the limited information on outcomes, difficulty in attributing SIF outcomes to the PSMA, and limitations on the availability of knowledgeable key informants. This does not suggest that these outcomes were not achieved, but rather that the evaluator was unable to make a judgment on this.
During the PSMA's implementation, the departments in which the SIF coordinating office resided were reorganized three times. The impact of reorganizing three different lead organizations during the implementation of a single government-wide initiative was significant. This also affected the evaluation. The reader should be mindful of the constraints on the evaluation imposed by these circumstances and the resulting data limitations. Because this evaluation took place after the close of the program, lessons learned rather than recommendations are presented.
The PSMA received royal assent on November 7, 2003, with the overall goal of modernizing employment and labour relations in the public service. The $238‑million PSMA Reserve Fund was establishedwithin the fiscal framework in order to implement the Act; initially, $38 million was used in support of early preparatory work and project management functions related to the PSMA's start-up.
In 2004, the remaining $200 million of the PSMA Reserve Fund was allocated to the SIF, which was established to guide investments in implementing the PSMA. The SIF was to be implemented in a way that:
A transitional government-wide body, the Public Service Modernization Act Secretariat (PSMA Secretariat), formerly the Human Resources Modernization Implementation Secretariat of the Treasury Board of Canada Secretariat, was created to orchestrate the changes brought about by the PSMA, coordinate efforts across departments and agencies, and manage the allocation of, and applications for, funds available for implementing the PSMA. The PSMA Secretariat, as part of the newly created Public Service Human Resources Management Agency of Canada (PSHRMAC), acted as the coordinating body or Project Office, with responsibility for the management of implementation funds, communications, and project oversight in implementing the PSMA.
In addition to PSHRMAC, the Public Service Commission (PSC), TBS, the Canada School of the Public Service (CSPS) and Justice Canada were directly involved in the implementation of the PSMA as Corporate Delivery Partners and had specific responsibilities for results.
The objective of this evaluation was to examine the implementation, relevance and performance (efficiency, effectiveness and economy) of the investments made under the SIF, the extent to which they facilitated achieving outcomes of the PSMA, and the lessons learned from this initiative. The scope of the evaluation focused on the departments that accessed SIF funds, which included central agencies. This evaluation did not assess the effectiveness of the tools or instruments implemented by these departments. Although the SIF was central to the PSMA's implementation, this was not an evaluation of the PSMA but of the use of SIF funds.[4]
During the planning phase of the project, the logic model was updated to ensure it reflected the full range of impacts of the SIF. The alignment of SIF activities and outputs with the approved PSMA outcomes was considered essential for developing appropriate indicators for the SIF that were consistent with the outcomes approved and monitored for the PSMA overall (see Figure 1.1). The short-term outcomes reflect the five notional SIF-funding envelopes, each with its own set of medium-term outcomes that link to the long-term outcomes for the SIF and the PSMA.
In accordance with the Treasury Board of Canada Secretariat's Policy on Evaluation, the evaluation used multiple lines of evidence and qualitative and quantitative data. The list below identifies the key evaluation issues and questions. These are addressed using four main lines of evidence: key informant interviews, document review, administrative data review, and case studies.
As noted above in 2003, when the PSMA Reserve Fund was established, the elements to be evaluated were identified. These questions focused on implementation rather than results. Because the evaluation was conducted between 2009 and 2010, the evaluation included an assessment of relevance in order to meet the requirements of the Policy on Evaluation and its related standards.
Efficiency
Economy
Effectiveness
* This refers to HRM broadly.
A total of 23 key informant interviews were conducted either in-person or by telephone, with representatives working in the area of human resources management in federal departments and agencies. The original methodology had planned for 50 interviews; however, difficulty in locating potential participants limited the number possible.
The document review was intended to address all evaluation questions. Three main types of documents were reviewed: background documents (e.g., the 2004 SIF Treasury Board Submission, the Master Plan for the PSMA Implementation Project); operational documents (e.g., business case guidelines for the SIF, business case evaluation templates, Human Resources Management Advisory Committee [HRMAC] minutes); and reporting and evaluation documents (e.g., progress reports to Treasury Board ministers, PSMA monitoring and reporting summary reports, and relevant program reviews).
The administrative data review built upon a number of separate spreadsheets provided by the project authority. Goss Gilroy Inc. amalgamated a spreadsheet (which included the department, name, number of employees, whether they were subject to the PSMA, whether they received PSMA funding, and whether they received additional support) and merged it with a large data file that contained a separate worksheet for each department. This worksheet included the information on the total approved expenditures under the PSMA Reserve Fund, actual PSMA Reserve Fund expenditures, and project costs funded from departmental reallocations. These items were recorded for each fiscal year for each of the notional funding envelopes of the SIF (as outlined in the preceding).
Five case studies were conducted to provide a more in-depth look at how SIF funds were used within departments and agencies. For each case study, the evaluator conducted at least one interview and reviewed the administrative data and documentary evidence for that department. The evaluation, however, did not evaluate the effectiveness of the tools, instruments or policies adopted or adapted by these departments for the SIF's implementation.
Case study summaries were appended to the Key Informant Interviews Technical Report.[5] Case studies were conducted with the following:
There were a number of limitations to the evaluation, which are described below. These limitations led to the evaluator being unable to assess the extent to which some intermediate outcomes and the ultimate outcome were achieved, which weakened conclusions on the extent of the achievement of immediate outcomes.
Availability of knowledgeable key informants: One of the main limitations of this evaluation was the low availability of key informants who were knowledgeable about SIF implementation in their departments. Only 23 key informants, or 46 per cent of the intended sample of 50 provided information, some of which was incomplete. Reasons for this included the following:
Limited information on outcomes: Departmental reporting was largely at the output level. Also, although information such as Management Accountability Framework (MAF) ratings was available for each department, the measures were at such a high level of aggregation that they could not be directly related to any one activity, let alone a specific SIF-funded project.
Departmental versus project impacts: One of the limitations of a horizontal evaluation is that departmental impacts cannot be determined with great accuracy. Although case studies were used as a line of evidence, this was not a departmental-level evaluation.
Attribution: The projects funded under SIF were implemented over several fiscal years, with different types of projects, alone or in combination with other sources of financing. The profiles of SIF projects, in terms of their nature, size and timing, were different for every department. In addition, the close relationship between the outcomes of the SIF and those of the PSMA added another level of complexity to attribution issues.[7]
The alignment of SIF outputs and activities with PSMA outcomes determined that any attribution of SIF funding to outcomes largely had to rely on case studies that were used to explore linkages between the SIF and outcomes at the departmental level.
Although there are limitations on the availability of knowledgeable key informants and limited information on outcomes, other lines of evidence were used to draw conclusions on the implementation of the SIF, strengthening the reliability of the validity of the results. Despite the limitations, the methodology meets the requirements of the Treasury Board's Policy on Evaluation and associated standards.
SIF activities were undertaken throughout departments and agencies to support the development of HRM capacity needed to implement the PSMA. The evidence showed that these activities were relevant. In fact, all key informants stated that there remains an ongoing need for the types of activities that the SIF funded.
Is there a continuing need to support departments and agencies in developing human resources management capacity?
The evaluation found evidence to suggest that there is a continuing need to support departments and agencies in developing HRM capacity.
The document review confirmed that having adequate HRM capacity was critical for implementing the HR reforms required by the PSMA and related legislation. The 2005 Status Report of the Auditor General cautioned that a lack of HRM capacity could "hinder the government's efforts to translate the provisions of the PSMA into concrete reforms"[8] and cause departments to "revert to their old systems, stalling progress toward modernized human resources management." PSHRMAC also identified in 2007 that HR capacity is a key challenge (among others) for implementing the PSMA.
The funding that the SIF provided to departments and agencies was meant to support them in addressing these challenges. There is also evidence that even after its sunset, there is a continuing need. For instance, the Prime Minister's Advisory Committee on the Public Service, in the Seventeenth Annual Report to the Prime Minister on the Public Service of Canada (2010), recognizes the "inadequacies of an outdated public service workplace"[9] and concludes that excellence in people management is fundamental and that role renewal is playing an important role.[10]
Almost all key informant interviews also indicated that there is a continuing need to support departments and agencies in developing HRM capacity. When questioned, they cited specific areas that still needed support, including training, human resources IT systems, staffing, and development of HRM tools and best practices. The document review shows that departments and agencies invested an additional $83.6 million of their own funds over the five-year period toward related projects. The evaluator considers this an indicator of their perceived need.
Did the objectives and related activities of the SIF generally align with the objectives and spirit of the PSMA?
The evidence was consistent in indicating that SIF objectives and activities were aligned with the objectives and spirit of the PSMA.
The specific objectives of the PSMA were as follows:
The SIF was specifically created to facilitate the PSMA's implementation. It was divided into notional funding envelopes (see Table 2.1) totalling approximately $200 million.
From the departmental perspective, almost all key informants who were interviewed indicated that the SIF was appropriately aligned with the spirit of the PSMA, demonstrated by the activities it funded to support departments and agencies in implementing PSMA legislation and related changes. A large majority of respondents indicated that the objectives and related activities of the SIF were well aligned with the needs of departments and agencies because they were, in general, broad enough and flexible.
Table 2.1. Strategic Investment Framework Notional Envelopes as of April 2004
Notional Envelope | Definition/Description | Notional Budget ($ millions) |
---|---|---|
Institutional Change | This funding supports the creation of the following:
|
60 |
PSMA HR IT Systems |
This funding supports adjustments to HR IT systems in relation to the following:
|
55 |
People Preparation | This funding is invested in the following:
People preparation also includes an amount for labour–management cooperative activities related to learning that are consistent with the intent of the PSMA. |
40* |
New Functions |
This is the funding to put in place resources for new functions where no base exists, e.g.:
|
36 |
Project Management |
This funding supports the horizontal management and related work to be undertaken, including the PSMA Secretariat. It also includes the following:
|
9 |
Total | $200 |
* The people preparation notional budget later increased to $62 million.
Efficiency of Strategic Investment Framework Implementation
The evaluation results indicate that the SIF was organized and implemented as expected. Although the Master Plan was issued a year later than the SIF, the first progress report indicated that the PSMA Secretariat developed and maintained project implementation timetables to identify key activities and critical milestones. Status and summary reports were used later on to inform the commitments and achievements. Evidence shows that all PSMA legislative requirements were met to a significant degree. There is evidence on the effectiveness of the project plan in guiding horizontal initiatives across departments and agencies.
How effective was the role of the Project Office, delivered in concert with other players?
How effective was the role of the Project Office in the context of the functionality of the governance model (i.e., as a secretariat)?
The evaluation focused on two main areas: the extent to which the role of the Project Officewas clear and adequately communicated, and the overall effectiveness of the Project Office.
The role of the Project Office was well articulated and communicated in SIF documentation; however, it is unclear to what extent it was well understood among departments and agencies. The evaluation found that the Project Office effectively managed the SIF's implementation. Evidence shows that the governance framework and management systems and processes in place for the SIF were adequate.
The document review found a number of sources that described the role of the Project Office. These included the PSMA Implementation Project Charter (2004), the first progress report on PSMA implementation (2005), the Results-Based Management and Accountability Framework for the PSMA Implementation Project (2005) and the Master Plan for the Implementation of the Public Service Modernization Act (2006).
However, the extent to which these documents were communicated to departments and agencies and other stakeholders is not clear. Although some respondents indicated their understanding of the role of the Project Office, most key informants reported a lack of clarity and/or confusion, citing lack of communication as the reason.
Progress reports and other assessments provided some evidence of the effectiveness of the Project Office in terms of overall management of the SIF's implementation. The progress reports prepared in 2005 and 2007 by PSHRMAC and the 2007 status report on PSMA deliverables provide sound evidence to confirm that the Project Office put in place the processes and mechanisms needed to effectively manage the SIF's implementation.
A 2009 assessment by the Treasury Board of Canada Secretariat of SIF management controls for adequacy and effectiveness, during the period from May 2004 and March 2009, found that the governance framework and management systems and processes that were put in place for the SIF were adequately designed.[15] It showed the following:
A few key informants indicated that the Project Management Committee was effective and kept everyone informed, although challenges were mentioned, including high turnover and the change in governance (when the Project Office was moved and then downsized).
How effective was the project plan and its implementation?
The evidence suggests that, for the most part, appropriate mechanisms were in place for effective implementation of the project plan.
The PSMA Implementation Project started in 2001 and was expected to sunset in 2006 but was extended until 2010. It encompassed four phases: conception and design; development and implementation; post-implementation; and monitoring, evaluation and reporting.
The PSMA Master Plan consisted of the Project Charter, project path, work stream plans, and project implementation timetables. The Master Plan included objectives of the PSMA's implementation, the governance landscape, governance and stakeholders, PSMA outcomes, risk management, the PSMA Reserve Fund, and the SIF funding life cycle.
The SIF funding life cyclewas a rigorous process used to manage SIF allocations. This process is summarized as follows:
Key guiding documents such as the SIF (April 2004), the Results-Based Management Accountability Framework (September 2005) and the PSMA Implementation Project Charter (November 2005) were completed and implemented. Progress reports and status reports subsequently provided summaries of completed activities, milestone dates, achievements, gaps and actions as required in the Master Plan 2006.
Progress reports on PSMA implementation indicated that delays in the implementation occurred as a result of departmental reorganizations and the time needed to manage the change agenda.
There is documentary evidence[17] that departments and agencies were provided with the PSMA-SIF business case guidelines and template to help prepare their business case(s) to access SIF funding. These clearly outlined the decision-making process for funding. Although some key informants suggested that the tools, templates and advice on costing could have been better, a large majority reported that SIF guidelines and criteria were clear.
Some feedback from key informants suggested that the template could have been simpler and less onerous to complete and that supporting documents could have better explained the criteria and provided advice on costing proposals. Key informants reported the issue of the short timeframe for implementation once the allocation decisions had been made. Key informants also reported issues regarding short timeframes between allocation decisions and implementation. Some departments and agencies were not ready to take advantage of SIF funds or had inadequate time for planning.
The SIF business case guidelines and template outlined five funding principles that were considered during the review of the business cases by the evaluation committee. The five principles were as follows:
Key informants were able to comment on only the horizontal partnership principle and generally indicated that the project plan was effective in coordinating collaboration across departments and agencies.
Table 2.2 provides a comparison of total SIF disbursements by notional funding envelope. Overall, $188.7 million, or 94 per cent, of SIF funding was disbursed to departments and agencies, which is an indication of an efficient approach. Over the five fiscal years, departments and agencies requested additional funds for training and learning initiatives through the business case process. As a result, $62.1 million of the $188.7 million was disbursed according to the "people preparation" envelope. When comparing the overall amount disbursed with the amount spent by departments and agencies in all envelopes, 83.4 per cent of the funds approved were invested by departments and agencies, which is again a key indicator of efficiency.
Table 2.2. Strategic Investment Framework Disbursements and Expenditures by Notional Envelope
Notional Envelope | SIF Envelope ($) | SIF Disbursements ($) | SIF Spending ($) | Percentage of SIF Disbursements | Percentage Spent of Approved Funding |
---|---|---|---|---|---|
New Functions | 36,000,000 | 29,611,036 | 32,450,948 | 82.3 | 109.6 |
HR Systems Changes | 55,000,000 | 46,913,298 | 45,155,097 | 85.3 | 96.3 |
People Preparation | 40,000,000 | 62,106,164 | 42,284,288 | 155.3 | 68.1 |
Institutional Change | 60,000,000 | 41,363,744 | 21,311,005 | 68.9 | 51.5 |
Project Management | 9,000,000 | 8,704,699 | 16,110,476 | 96.7 | 185.1 |
Total | 200,000,000 | 188,698,940 | 157,311,814 | 94.3 | 83.4 |
Source: PSMA-SIF database
The audit and evaluation component for the implementation of the PSMA (including the SIF) required the development of an evaluation framework, terms of reference for the evaluation, a results-based management and accountability framework, and reporting. The latter was based on progress reports.
Each fiscal year, a summary of department/agency reports on PSMA implementation (PSMA monitoring and summary reports, including progress and financial data) was presented to PSHRMAC, from 2005–06 to 2008–09. PSMA monitoring and reporting reports for fiscal year 2008–09 noted, however, that the reporting burden on departments and agencies was a concern. To address this, call letters replaced the HR Integrated Reporting Portal and the non-financial data requested of departments and agencies. The revised approach was limited to brief descriptions of SIF activities, which were output-based and therefore did not provide outcome information.
PSHRMAC used the data collected through this reporting process to draft progress reports and a summary of results. Because the results reporting was output-oriented, it could not be used to assess outcomes.
All business cases submitted for SIF funding consideration were required to address risk management and include an assessment of significant risk areas. Risk assessment and risk management were built into the assessment of funding applications. The SIF business case guidelines, short evaluation form and the SIF business-case detailed evaluation template are examples of the tools developed to assess risk in each business case.[19]
How effective was the project plan in guiding horizontal work across departments?
The documentary evidence indicates that the project plan was effective in guiding horizontal work across departments and agencies. Most interviewees, however, were unable to comment on this.
The Management Framework[20] component of the PSMA implementation, established as an objective "the development and implementation of stakeholder engagement throughout the process." Under this framework, the governance landscape component involved "the description of the governance structure which included an interdepartmental committee structure consisting of central agencies and departmental representatives to contribute to the implementation of the PSMA." The support of horizontal management and related work undertaken by either the PSMA Secretariat or the departments and agencies were set out under the notional envelope "project management." Horizontal partnership was also one of the funding principles considered in evaluating business cases.[21]
The document review shows that two interdepartmental committees were created to support horizontal collaboration and partnership. The Interdepartmental Project Management Committee was an advisory body composed of 14 representatives, and the Interdepartmental Project Management Network was a community forum composed of representatives from 90 organizations.
The second progress report provides some evidence on the activities undertaken by the PSMA Secretariat that could be linked to horizontal partnership activities:
The data review shows that functional communities and federal councils accessed SIF funds through some departments acting as "bankers." Approximately $2.2 million (3.3 per cent) of the $65.4 million received by departments and agencies was used by functional communities and federal councils for online tools and learning events.
Based on information provided by key informants, the project plan was not intended for distribution to departments and agencies. This seems to be reflected by the fact that most respondents could not comment on the effectiveness of the project plan, which was therefore not a conclusive line of evidence. Nonetheless, a few respondents indicated that interdepartmental committees, which included small and large departments and agencies, facilitated coordination across the federal government. Only a few commented that departments and agencies worked collaboratively.
Several respondents pointed out that there was not enough emphasis on assisting smaller departments and agencies, suggesting that the plan may have been too restrictive for smaller agencies and provided limited opportunity for them to collaborate horizontally with larger departments and agencies at the planning stages.
On the other hand, the SATST final report of 2009[22] indicates that 100 per cent of clients surveyed in the SATST 2008 survey agreed with the statement "the support and assistance that SATST provided my organization is of significant assistance in integrating the requirements of the Public Service Modernization Act."
How effective was the IT capability and functionality specific to this project?
There appears to have been sufficient IT capability and functionality for SIF implementation as evidenced by the creation of a combined PSMA-SIF database and the HR Integrated Reporting Portal. However, there were no users available to interview, which would have provided another line of evidence to support this conclusion.
The document review indicated that the PSMA-SIF database was developed in 2007 to ensure stewardship of the PSMA Reserve Fund and therefore of SIF funds. The electronic system reconciled and accounted for all funding to ensure the tracking and monitoring of expenditures, and to record information on the internal reallocations made by departments and agencies. It also facilitated in preparing reports that were considered to be timely, accurate, up to date and in line with good financial management practices. In fiscal year 2006–07, the HR Integrated Reporting Portal was created to gather SIF data from departments and agencies. Key informants were not able to speak to the IT capability or functionality related to SIF funds.
To what degree were PSMA legislative requirements met?
The evaluation found that PSMA legislative requirements were met to a significant degree.
The document review found that all legislative components of the PSMA are in force. This includes the Public Service Labour Relations Act (April 1, 2005), amendments to the Financial Administration Act (April 1, 2004), a new Public Service Employment Act (December 31, 2005), and amendments to the Canadian Centre for Management Development via the Canada School of Public Service Act (April 1, 2004).
As well, all institutional changes were in place at the time of the second progress report (2007): mandates were revised for the Public Service Commission of Canada, Treasury Board of Canada Secretariat and the Privy Council Office; the Canada School of Public Service was launched; the Public Service Staffing Tribunal and the Public Service Labour Relations Boardwere operational; and PSHRMAC was created to lead the PSMA Implementation Project.
The evaluation did not assess the degree of compliance by departments and agencies that had PSMA legislative requirements. However, evidence from the document review suggested that additional requirements stemming from the legislative package were met. For instance, all staffing delegation instruments were in place government-wide; departmental staffing policies, programs and processes had been developed; and departments and agencies had reviewed and updated labour relations delegation instruments and grievance procedures to be in line with changes to the Financial Administration Act and Public Service Labour Relations Act.
Were there appropriate linkages and leveraging with non-PSMA modernization activities that contributed to the successful implementation of the PSMA?
Although leveraging generally took place, there was insufficient evidence to clearly indicate the extent to which linkages and leveraging with non-PSMA modernization activities occurred or were even the objective of SIF activities.
A large majority of respondents indicated that non-PSMA modernization activities occurred parallel to, and supported, SIF implementation. However, there were insufficient details to clarify the extent to which this occurred.
A few respondents reported having made some effort to link non-PSMA and non-SIF implementation activities. Those who did provided examples referred to shared services, the implementation of PeopleSoft, and the development of classification tools. These activities enabled departments and agencies to be more efficient overall and contributed to improved results related to SIF implementation.
Was there value for money of the actual investment?
Although results from the document review and from key informant interviews showed that SIF funding did indeed provide value, the evaluation could not determine if this could have been accomplished with fewer resources.
A large majority of key informants indicated that without SIF funds, their departments and agencies would not have been able to achieve the changes required by the PSMA as successfully or as quickly, and most of the smaller departments and agencies would not have had the resources at all. The initiatives reportedly made departments and agencies more effective and efficient in the area of HRM.
SIF investment provided opportunities to undertake activities such as:
The above suggests that SIF funding provided value. However, since alternatives to the SIF were not identified and costed, the evaluation is not able to make a conclusion regarding value for money.
What was the nature and level of uptake of institutions?
The evidence shows that the level and nature of uptake of SIF funds was appropriate. Eighty-eight per cent of departments and agencies were assisted either directly or indirectly through SATST or parent departments with their PSMA implementation requirements.
In 2009, there were 115 departments and agencies in the administrative database. Table 2.3 indicates the coverage of departments and agencies by SIF projects.
Table 2.3. Coverage of Department and Agencies by Strategic Investment Framework Projects 2008–09
Institution | Number of Departments and Agencies | Percentage of Departments |
---|---|---|
Direct funding (includes departments, agencies, the Treasury Board Portfolio and Treasury Board Partners) | 32 | 28 |
Direct funding (also received SATST support) | 9 | 8 |
SATST support (received no direct funding but received funding through PSHRMAC) | 43 | 37 |
Indirect support through parent department | 17 | 15 |
No funding; participated in government-wide or functional events | 14 | 12 |
Total* | 115 | 100 |
Functional organizations and Regional Councils | 10 | — |
*In 2005, the database showed 117 organizations. By 2009, the Law Commission and the Canadian Centre for Independent Resolution of First Nations Claims no longer existed.
The administrative data review also found that departments and agencies that received direct funding under the SIF accounted for more than 90 per cent (180,000) of employees in the core federal public service. Departments and agencies that received indirect support through parent departments and support from SATST accounted for 2 per cent of the employees (see Table 2.4).
Table 2.4. Coverage of Employees by Strategic Investment Framework Projects 2008–09
Institution | Number of Departments and Agencies | Core Public Administration FTEs* | Separate FTEs | Total FTEs |
---|---|---|---|---|
Direct funding (includes departments, agencies, the Treasury Board Portfolio and Treasury Board Partners | 32 | 184,158 | 1,639 | 185,797 |
Direct funding (also received SATST support) | 9 | 1,454 | 115 | 1,569 |
SATST support (received no direct funding but received funding through PSHRMAC) | 43 | 3,267 | 2,387 | 5,654 |
Indirect support through parent department | 17 | 640 | 1,595 | 2,235 |
No funding; participated in government-wide or functional events | 14 | 13,230 | 59,088 | 72,318 |
Total** | 115 | 202,749 | 64,824 | 267,573 |
Functional organizations and Regional Councils | 10 | — | — | — |
*Full-time equivalents
**In 2005, the database showed 117 organizations. By 2009, the Law Commission and the Canadian Centre for Independent Resolution of First Nations Claims no longer existed.
Of the 41 organizations that received SIF funding, 33 were line departments or part of the five PSMA Corporate Partners Groups (Justice Canada, the Public Service Staffing Tribunal, the Public Service Labour Relations Board, the Public Service Commission of Canada and the Privy Council Office). Departments and agencies received $65.4 million, with approximately $164 million awarded to Treasury Board Portfolio organizations and PSMA Corporate Delivery Partners.
Economy was demonstrated by the significant leveraging achieved through departmental reallocations to projects funded through the SIF (see Table 2.5). Over $83.5 million was reallocated by departments, agencies and central partners from 2004 to 2009, which amounted to approximately 30 per cent of the total PSMA expenditures.
Table 2.5. Total Approved Funding and Expenditures*
Year | PSMA Reserve Fund | Internal Reallocations Reported | Total Expenditures | |
---|---|---|---|---|
Disbursements | Reported Expenditures | |||
Before the SIF | ||||
2002–03 | $16,725 | $15,418 | 0 | 15,418 |
2003–04 | $23,954 | $16,678 | 0 | 16,678 |
Total Before the SIF | $40,679 | $32,096 | 0 | 32,096 |
After the SIF | ||||
2004–05 | $14,198 | $14,114 | 8,520 | 22,634 |
2005–06 | $81,903 | $56,499 | 42,032 | 98,531 |
2006–07 | $25,302 | $18,530 | 12,067 | 30,597 |
2007–08 | $37,340 | $33,639 | 13,943 | 47,581 |
2008–09 | $29,957 | $34,530 | 7,003 | 41,533 |
Total after the SIF | $188,699 | $157,312 | 83,566 | 240,877 |
Total PSMA-SIF Fund** | $229,378 | $189,408 | 83,566 | 272,973 |
Note: Of the total $238-million PSMA Reserve Fund, $8.6 million was returned to the fiscal framework by way of the February 2008 federal budget.
* In $000s*
*Totals may not add due to rounding.
Source: SIF, Summary data from database, September 2009
Disbursements were made for activities under each notional envelope (see Table 2.6). Over time, there was substantial use of SIF funding for all types of projects, even within the last years of the funding. The expenditures under the SIF were approximately $15 million per year prior to 2005–06 but peaked at $56.4 million in 2005–06. Although the use of the fund dropped in subsequent years, expenditures still averaged approximately $29 million per year.
Table 2.6. Strategic Investment Framework Expenditures by Envelope (Thousands) and Fiscal Year
Envelopes | Pre-SIF | Post-SIF | |||||
---|---|---|---|---|---|---|---|
2002–03 | 2003–04 | 2004–05 | 2005–06 | 2006–07 | 2007–08 | 2008–09 | |
New Functions | — | — | $2,709 | $8,267 | $8,390 | $7,436 | $5,646 |
HR System Changes | $1,500 | $1,018 | $2,319 | $12,768 | $8,243 | $7,470 | $14,352 |
People Preparation | — | — | $1,994 | $22,637 | $1,631 | $8,571 | $7,449 |
Institutional Change | $80 | $1,478 | $3,736 | $6,089 | $100 | $6,026 | $5,358 |
Project Management | — | — | $3,354 | $6,735 | $164 | $4,134 | $1,721 |
Not Separated by Envelope | $13,837 | $14,181 | — | — | — | — | — |
Total* | $15,417 | $16,677 | $14,113 | $56,499 | $18,530 | $33,638 | $34,529 |
*Totals may not add due to rounding.
Source: SIF, Summary data from database, 2009
The evaluation could not conclude firmly on the extent to which the SIF investment strategy contributed to achieving the expected medium- and long-term outcomes of the PSMA. Nonetheless, the expected outputs and most immediate and short-term outcomes were achieved.
How effective was the application of the investment strategy in achieving the desired objectives (results and outcomes) of the PSMA?
The application of the SIF investment strategy appears to have been effective in facilitating the achievement of the PSMA objectives. However, due to limited lines of evidence, it is unclear to what extent this was the case.
Assessing the SIF investment strategy's effectiveness in achieving the PSMA objectives can be done by looking at a combination of factors, the key ones being as follows:
At the outset, a targeted rather than broad funding approach was used. Envelopes were dedicated to agreed upon categories that supported key elements of the PSMA (e.g., people preparation, institutional change). The distribution of funding across the notional envelopes was fairly even (with the exception of project management, which represented the central implementation unit), which demonstrates that they all were relevant to departments' and agencies' needs. Using leveraging as a tool in the funding process helped create a more economical means of implementation. The use of "banker" departments helped reach functional communities horizontally. Using the SATST as a mechanism for representing small departments and agencies and funding distribution also assisted in the efficient disbursement of funding.
The oversight mechanisms used by the Project Office helped ensure that funds were being used for the agreed upon activities. However, as previously indicated, results reporting was limited and output-based. The PSMA-SIF database, the HR Integrated Reporting Portal, and call letters to departments and agencies provide evidence that projects were appropriately monitored and reported. Compliance with the call letters among departments and agencies was high for each fiscal year (for which data was available): 100-per-cent compliance in 2005–06, 95-per-cent compliance in 2006–07, and 100-per-cent compliance in 2007–08.[23]
Finally, evidence from the document review, key informant interviews and case studies suggest that the investment strategy of the SIF created to some extent a more flexible staffing framework, fostered more collaborative labour management relations, and clarified accountability for deputy heads and managers.
What results were achieved (including both outputs and outcomes)?
Information analyzed from the five case studies and key informant interviews demonstrate that all SIF outputs and most intended immediate and short-term outcomes[24] were to some extent achieved.
Although the evidence was stronger in some areas than in others, key informants mentioned a wide range of projects that resulted from SIF implementation. Importantly, most key informants indicated that several project outcomes could be attributed to the SIF given that the projects could not have gone forward without funding.
The document review found evidence to indicate that all expected outputs were achieved:
Of the three immediate outcomes, two were achieved to some extent.
Although some key informants could not comment on awareness of the SIF and its guidelines, it could be inferred that most of the departments and agencies were familiar with these documents because they successfully accessed funding.
As previously indicated under "Relevance," the SIF investment strategy included the submission of business case proposals, which were based on specific funding principles and the business case guidelines and template. These procedures ensured that proposals from departments and agencies were aligned with SIF principles and PSMA priorities. Evidence already presented indicates that this outcome was achieved.
The administrative data review found that departments and agencies co-funded SIF projects or that they were funding other PSMA-related activities while in receipt of SIF funding. On average, reallocations made by departments and agencies accounted for $83.5 million, or 30 per cent, of total PSMA expenditures.
Whether projects were implemented in a timely manner is not clear; however, the evidence suggests that timeliness was an issue. The administrative data review found that over three quarters of departments and agencies lapsed SIF funds from one year to the next and occasionally over two or more years. Key informants indicated that delays in project implementation were frequently a result of –09 because of delays in PSMA implementation as a result of the reorganization of the management office.
The document review showed that Treasury Board ministers were informed periodically on the progress of the PSMA's implementation, including SIF funding investments and the results achieved through progress reports.
Although the evidence shows that all six short-term outcomes were achieved, it was not adequate in order to verify the extent of achievement.
Different progress reports confirmed that the institutional changes planned with SIF funds were successfully achieved, such as the following:
Key informants were generally unable to comment on enhancements made to existing government-wide IT systems (e.g., introduction of PeopleSoft, Human Resources Information System, or HRIS), but they did identify HR IT systems as a top priority area that requires continuing support.
The document review found a number of examples of HR IT systems that were developed to support SIF implementation, including the PSMA-SIF database and the HR Integrated Reporting Portal. At the departmental level, the case studies showed that a number of specific systems were also supported by the SIF. For instance, Infrastructure Canada created an electronic information system (HRIS) to replace a paper-based one to deal more effectively with HR issues. National Defence established an integrated staff log application to manage and report on large collective processes, and Agriculture and Agri-Food Canada made enhancements to its business intelligence tools.
The 2006–07 summary report indicated that by 2007, 75 per cent of reporting organizations conducted activities in the area of learning and communication such as the following:
Results from the document review and case studies found a number of examples of learning events on various PSMA-related topics that took place between 2005 and 2008, such as refresher training on various HR-related activities, policy training for HR professionals, and sub-delegation training for managers. Actual participation rates for all departments and agencies were not available.
Considering that all departments and agencies directly or indirectly used SIF funds and that almost all were engaged in PSMA activities, including learning and communicating, it can be inferred that public service managers, HR professionals and staff have acquired to some degree the skills and knowledge to implement the PSMA. However, the evaluation could not assess the extent to which this is the case.
Findings from interviews and case studies indicate that almost all key informants agreed that public service managers have thenecessary skills and knowledge to implement the PSMA.
Evidence gathered from the document review and case study analysis indicates that several departments and agencies established or adopted new functions. However, at the time of data collection there was no clear indication that these systems became operational or had secured A-base funding. The document review shows that in several departments and agencies, HR planning was integrated with business planning, and new integrated HR planning tools were developed and are in use (at Agriculture and Agri-Food Canada, Fisheries and Oceans Canada, Environment Canada, Industry Canada, Aboriginal Affairs and Northern Development Canada, and Public Works and Government Services Canada).[26] Examples from the case studies show that human resources planning at Agriculture and Agri-Food Canada was aligned with the elements of the "people" component of the MAF, whereas at National Defence human resources planning was aligned with the National Defence's Appointment Delegation and Accountability Instrument. Other new functions included the development of instruments for guidance on human resources topics, such as the "Staffing Desk Reference for Managers" at National Defence.
Horizontality was one of the SIF's funding principles and, based on the document review, this was achieved in terms of the role of the Project Office and governance structures. Most key informants believed that the project management was somewhat effective in achieving horizontal PSMA implementation.
The outcomes that were expected from SIF funding in the medium-term included the following:
Although these outcomes may have been achieved, the limitations relating to reporting and stakeholder access for interviews resulted in the evaluation being unable to conclude on them.
The outcomes that were expected to be achieved in the long-term as a result of SIF funding were as follows:
As with the intermediate outcomes, the long-term outcomes may have been achieved; however, the evaluation could not conclude that this was the case.
SIF-sponsored activities were undertaken throughout departments and agencies to support the development of HR management capacity needed to implement the PSMA. The evidence showed that these activities were relevant. In fact, all key informants stated that there remains an ongoing need for the types of activities that SIF funded.
Evaluation questions relating to efficiency were mainly focused on how effectively the initiative had been implemented. In this regard, the evaluation results indicate that the SIF was organized and implemented as expected. Ninety-four percent of SIF funding was disbursed to departments and agencies, and 83.4 per cent of the funds approved were invested by departments and agencies.
Although the Master Plan[27] was issued a year later than the SIF, the first progress report indicated that the PSMA Secretariat developed and maintained project implementation timetables to identify key activities and critical milestones. Status and summary reports were used later on to inform the commitments and achievements. Evidence shows
The evaluation could not make a firm conclusion on the extent to which the SIF investment strategy contributed to achieving the medium- and long-term outcomes of the PSMA, in part because of the lack of informants and reduced reporting among funding recipients.
Nonetheless, the expected outputs and most immediate and short-term outcomes were achieved. Outputs such as guidelines, communications and training led to the immediate outcomes, thus resulting in departments applying and receiving funding for projects that aligned with the PSMA priorities and principles, although timeliness of implementation was identified as a potential issue. The short-term outcomes, for the most part, also appear to have been achieved as evidenced by institutional change[28], HR IT systems, people preparation events, and new HR functions to support PSMA implementation. However, the evaluation was unable to conclude on the extent to which these, in turn, supported public service HR needs, resource processes, cultural change and accountabilities. Similarly, conclusions could not be drawn regarding the long-term outcomes of hiring the right people, collaborative labour–management relations, increased focus on learning and training for employees at all levels, and improved clarity in roles and accountability—a significant limitation of the evaluation.
Many of the lessons learned from this evaluation relate to the issue of performance measurement and reporting. Although the reorganizations of PSHRMAC and the CPSA highlighted the issue of information management, greater awareness is needed more generally regarding the importance of establishing performance measurement strategies and reporting frameworks in advance of implementing government initiatives in order to provide senior management with the information needed on the impact of large initiatives.
[1]. In 2003, when the PSMA Reserve Fund was established, there was a requirement to evaluate only the use of SIF funds (2004–08) rather than the whole PSMA Reserve Fund. This requirement was confirmed in 2008.
[2]. According to the Master Plan for implementing the PSMA (2006), the plan was conceived as "a strategic, rather that operational plan. It sets out the overall framework for delivering on the promise of PSMA in a complex, ever-changing environment, with multiple stakeholders, varying capacities, diverse accountabilities and significant challenges."
[3]This included: the establishment of Public Service Staffing Tribunal, the Public Service Labour Relations Board and the Canada School of Public Service, as well as changes to existing institutions such as the Public Service Commission of Canada. Also see page 20.
[4]. In 2003, when the PSMA Reserve Fund was established, there was a requirement to evaluate only the use of SIF funds (2004–08) rather than the whole PSMA Reserve Fund. This requirement was confirmed in 2008.
[5]. The Key Informant Interviews Technical Report summarizes information gathered from interviews with key informants.
[6]. The SATST was established in 2005 through the SIF to help small agencies implement the PSMA.
[7]. The PSMA implementation costs were financed through an initial allocation for preparatory work and management, the SIF Fund, and internal reallocations by departments and agencies. Costs of outputs were reported by departments and agencies on the total PSMA implementation cost rather than the SIF, making it difficult to attribute outputs to the SIF only.
[8]. Status Report of the Auditor General of Canada, February 2005
[9]. Seventeenth Annual Report to the Prime Minister on the Public Service of Canada, p. 6
[10]. Seventeenth Annual Report to the Prime Minister on the Public Service of Canada, p. 46
[11]. The Public Service Staffing Tribunal has the authority to deal with complaints related to internal appointments, layoffs, the implementation of corrective measures ordered by the Tribunal, and revocations of appointments.
[12]. A new mandate of the Public Service Labour Relations Board has been proclaimed, which includes an expansion of the existing Public Service Staff Relations Board (compensation research function and increased mediation services).
[13]. The merger of the Canadian Centre for Management Development with Training and Development Canada led to the establishment of the new Canada School of Public Service, including transfer of learning resources from the Public Service Commission of Canada for language training.
[14]. The effectiveness of implementation is a different issue than that of effectiveness of a program in meeting its expected outcomes.
[15]. Because of insufficient documentation on file, the assessment was unable to determine the consistent application of select key management controls.
[16]. Key informants indicated that Memorandums of Understanding were not signed with all participating departments and agencies.
[17]. PSMA-SIF business case guidelines and template, 2007.
[18]. PSMA-SIF business case guidelines and template, 2007, p. 6
[19]. For example, the SIF business-case guidelines short evaluation form and the SIF business-case detailed evaluation template were used.
[20]. Master Plan, PSMA, PSHRMAC, 2006
[21]. That is, to encourage PSMA initiatives that will lead to service-wide benefits, or enable shared, cumulative benefits across multiple organizations with a view to gaining efficiency and in meeting collective needs. Organizations were encouraged to seek out and pursue horizontal partnerships with other departments and agencies.
[22]. Small Agencies Transition Support Team (SATST) Final Report, March 31, 2009.
[23]. Summary of results documents for fiscal years 2005–06, 2006–07 and 2007–08.
[24]. Short- and medium-term outcomes in the logic model are organized by the SIF notional funding envelope.
[25]. A-base secure funding is sustainable funding that allows for the implementation of projects and initiatives.
[26]. "PSMA—Summary of Results from HR Portal (FY 2006–07)," October 3, 2007.
[27]. According to the Master Plan for the Implementation of the Public Service Modernization Act (2006), the document was conceived as "a strategic, rather that operational plan. The Master Plan set out the overall framework for delivering on the promise of PSMA in a complex, ever-changing environment, with multiple stakeholders, varying capacities, diverse accountabilities and significant challenges."
[28]This included: the establishment of Public Service Staffing Tribunal, the Public Service Labour Relations Board and the Canada School of Public Service, as well as changes to existing institutions such as the Public Service Commission of Canada. Also see page 20