ARCHIVED - Evaluation of the Public Service Modernization Act Strategic Investment Framework

• Previous Page
• Table of Contents
• Next Page

This page has been archived.

Executive Summary

This report presents the findings of an evaluation of the Strategic Investment Framework (SIF), which was a fund established to guide investments in implementing the Public Service Modernization Act (PSMA).

The objective of this evaluation was to examine the implementation, relevance and performance (efficiency, effectiveness and economy) of the investments made under the SIF, the extent to which they facilitated achieving outcomes of the PSMA, and the lessons learned from this initiative. The scope of the evaluation focused on the departments that accessed SIF funds, which included central agencies. This evaluation did not assess the effectiveness of the tools or instruments implemented by these departments. Although the SIF was central to the PSMA implementation, this was not an evaluation of the PSMA but of the use of SIF funds.^[1]

Between 2004 and 2009 $200 million in SIF funding was provided to 40 departments and agencies to support the achievement of the PSMA's objectives, which were as follows:

• Create a more flexible staffing framework to manage and support employees and to attract the best people, when and where they were needed;
• Foster more collaborative labour–management relations to ensure a healthy and productive workplace; and
• Clarify accountabilities for deputy heads and managers.

The funds were applied to five notional envelopes: institutional change, PSMA human resources (HR) information technology (IT) systems, people preparation, new functions, and project management.

The methodology for the evaluation incorporated the following lines of evidence:

• Document review;
• Administrative data review;
• Five departmental case studies; and
• 23 key informant interviews.

A key feature of this evaluation was that this initiative had ended prior to the evaluation's implementation. As a result, rather than recommendations, lessons learned are presented that could be applied to future funding programs that have a similar design. In addition, the legislative review of the PSMA was undertaken at a later time, during which the environment evolved and, therefore, may have arrived at additional insights and/or conclusions.

A number of limitations, including the timing of the evaluation after the initiative ended, resulted in restrictions on the availability of data to support the evaluation. For instance, the department in which the SIF Project Office was situated was reorganized three times during the life of the initiative, which resulted in the inability to locate many knowledgeable potential informants. This also meant that a survey could not be undertaken.

The SIF appears to have been well implemented, resulting in outputs being created to appropriately guide SIF funding to projects that were consistent with short-term outcomes. There was sufficient evidence for the evaluation to make conclusions on immediate and short-term outcomes but not enough to make conclusions on medium- and long-term outcomes. This should not be interpreted to mean that these outcomes were not achieved, but rather that they could not be measured or assessed based on the available evidence.

Conclusions

Relevance

SIF-sponsored activities were undertaken throughout departments and agencies to support developing human resources management (HRM) capacity needed to implement the PSMA. The evidence showed that these activities were relevant. In fact, all key informants stated that there remains an ongoing need for the types of activities that were funded by the SIF:

• The evaluation found evidence to suggest that there is a continuing need to support departments and agencies in developing HRM capacity.
• The evidence was consistent in indicating that SIF objectives and activities were aligned with the objectives and spirit of the PSMA.

Efficiency

Evaluation questions relating to efficiency focused mainly on how effectively the initiative had been implemented. In this regard, the evaluation results indicate that the SIF was organized and implemented as expected. Ninety-four per cent of SIF funding was disbursed to departments and agencies, and 83.4 per cent of the funds approved were invested by departments and agencies.

Although the Master Plan^[2] was issued a year later than the SIF, the first progress report indicated that the PSMA Secretariat developed and maintained project implementation timetables to identify key activities and critical milestones. Status and summary reports were later used to inform the commitments and achievements. Evidence shows that all PSMA legislative requirements were met to a significant degree. There is evidence on the project plan's effectiveness in guiding horizontal initiatives across departments and agencies:

• The role of the Project Office was well articulated and communicated in SIF documentation; however, it is unclear to what extent it was well understood among departments and agencies. The evaluation found that the Project Office effectively managed the SIF's implementation.
• The evidence suggests that mechanisms were in place to implement the SIF project plan effectively.
• Documentary evidence indicates that the project plan was effective in guiding horizontal work across departments and agencies. Most interviewees, however, were unable to comment on this.
• There appears to have been sufficient IT capability and functionality for the SIF's implementation. However, there were no users available to interview, and this would have provided another line of evidence to support this conclusion.
• The evaluation found that PSMA legislative requirements were met to a significant degree.
• Although leveraging generally took place, there was insufficient evidence to clearly indicate the extent to which linkages and leveraging with non-PSMA modernization activities occurred or were the objective of SIF activities.

Economy

Although results from the document review and from key informant interviews showed that SIF funding did indeed provide value, the evaluation could not determine if this could have been accomplished with fewer resources.

The evidence shows that the level and nature of uptake of SIF funds was appropriate. Eighty-eight per cent of departments and agencies were assisted either directly or indirectly through the Small Agencies Transition Support Team (SATST) or parent departments with their PSMA implementation requirements.

Effectiveness

The evaluation could not make a firm conclusion on the extent to which the SIF investment strategy contributed to achieving medium- and long-term outcomes of the PSMA, in part because of a lack of informants and reduced reporting among funding recipients.

Nonetheless, the expected outputs and most immediate and short-term outcomes were achieved. Outputs such as guidelines, communications and training led to the immediate outcomes, thus resulting in departments applying and receiving funding for projects that aligned with PSMA priorities and principles, although the timeliness of implementation was identified as an issue. The short-term outcomes, for the most part, also appear to have been achieved, as evidenced by institutional change^[3], HR IT systems, "people preparation" events, and new HR functions to support PSMA implementation. However, the evaluation was unable to draw a conclusion on the extent to which these, in turn, supported public service HR needs, resource processes, cultural change and accountabilities. Similarly, conclusions could not be drawn regarding the long-term outcomes of hiring the right people, collaborative labour–management relations, increased focus on learning and training for employees at all levels, and improved clarity in roles and accountability. These represent a significant limitation of the evaluation.

• The application of the SIF investment strategy appears to have been effective in facilitating the achievement of PSMA objectives; however, due to limited lines of evidence, it is unclear to what extent this was the case.
• Information analyzed from the five case studies and key informant interviews demonstrate that all SIF outputs and most intended immediate and short-term outcomes were achieved to some extent.
• Although the evidence shows that short-term outcomes were achieved, it was too limited to be definitive regarding the extent to which this was the case.

Lessons Learned

Many of the lessons learned from this evaluation relate to the issue of performance measurement and reporting. Although the reorganizations of PSHRMAC and the CPSA highlighted the issue of information management, greater awareness is needed regarding the importance of establishing performance measurement strategies and reporting frameworks before implementing government initiatives. Such strategies and frameworks would provide senior management with the information needed on their impact.

• Large funding initiatives such as the SIF should, whenever possible, be implemented from within a stable organizational structure. Reorganizations that take place simultaneously with implementing a large initiative run the risk of preventing it from achieving its intended outcomes, or of disrupting key elements from being executed effectively and efficiently. If this cannot be avoided, an information management strategy specific to the reorganization initiative, notwithstanding the organizational location, should be developed that includes all the relevant contacts for the initiative's specific responsibilities.
• Reporting frameworks should be designed to balance expenditure and outcome (not just output) reporting with the administrative demand. These frameworks should be designed in consultation with all stakeholders well in advance of implementation.
• Performance measurement strategies need to be developed before implementing initiatives in order to provide the information needed for evaluations.
• Conducting a baseline assessment at the initial stage of the implementation of an initiative would provide program managers with a basis for comparing results that could measure the impacts of a policy, program or initiative.
• When planning for an evaluation, consideration should be given to evaluating all related funds that contribute to a single outcome. In this case, for instance, the PSMA Reserve Fund ($238 million) included the SIF ($200 million), both of which contributed to the same desired outcomes. It is very difficult to isolate the impact of the $38 million used for the start-up phase from SIF funds, since they were all part of the PSMA implementation.
• Developing a system to follow up on initiatives after they end will help program managers track the long-term benefits of long-term initiatives and their implementation mechanisms such as the PSMA and SIF.

• Previous Page
• Table of Contents
• Next Page