ARCHIVED - IM/IT Investment Evaluation Guide

• Previous Page
• Table of Contents
• Next Page

This page has been archived.

Evaluation Phase: Learn From the Process

3.1 Evaluation Processes

The Evaluation phase "closes the loop" on the IM/IT investment management process by comparing actuals against estimates in order to assess performance and identify areas where future decision making can be improved. Lessons that are learned during the Evaluation phase should be geared towards modifying future Selection and Control decisions. Central to this process is the post-implementation review with its evaluation of the historical record of the project.

3.1.1 Conducting Post-implementation Reviews (PIRs) Using a Standard Methodology

Once a project has reached a final end point (e.g., the project is fully implemented, the project has been cancelled, etc.), a post- implementation review (or post-investment review) should be conducted. This review will usually occur about 3 to 12 months after a project has reached its final end point and should be conducted by a group other than the project development team to ensure that it is conducted independently and objectively.

Organizations often spend significant time and resources focused on selecting IM/IT projects, but less attention is given to evaluating projects after they are implemented. Yet the information gained from PIRs is critical for improving how the organization selects, manages, and uses its IM/IT resources.

Each PIR that is conducted should have a dual focus - it should:

1. provide an assessment of the implemented project, including an evaluation of the development process, and
2. indicate the extent to which the organization's investment decision-making processes are sustaining or improving the success rate of IM/IT projects. The following are three essential areas that should be evaluated as part of a complete PIR:

• a) Customers : Surveys should be conducted to determine users' satisfaction with the end product. There should also be a focused look at how well the project supports the organization's various business processes. Many of the intangible benefits that were identified at the outset will relate to how customers and end users feel about the final project.

• b) Mission/Program Impact : A close look should be taken to determine whether the implemented system has achieved its intended impact, and whether this impact is still aligned with mission goals. An assessment should also be made of other project-specific aspects, such as an estimate of cost savings that have been achieved, compliance with the information technology architecture, evaluations of the information product (accuracy, timeliness, adequacy, and appropriateness of information), and identification of additional maintenance or security issues.

• c) Technical Capability: Finally, an evaluation should be made of the technical aspects of the project, both current and future. Such an evaluation may focus on such factors as the competency of the workforce to use the new system and employee satisfaction or retention, the extent to which advanced technology was used, and the methodological expertise of the development team.

To ensure that each project is evaluated consistently, the organization should have a documented methodology for conducting PIRs. This methodology, which should be used at all organizational levels, should spell out roles and responsibilities for conducting reviews and for taking actions based on the results. PIRs should be required on a regular basis to ensure that completed projects are reviewed in a timely manner. The organization should also have policies or procedures that document how information from the PIRs is to be relayed back to decision makers.

Finally, because there is a great deal of knowledge that can be gained from failed projects, evaluations should also be conducted for projects that were cancelled prior to being fully implemented. Although project accountability is important, these evaluations should focus on identifying what went wrong with the project, in order to learn from mistakes and minimize the chances of their being repeated.

3.1.2 Feeding Lessons Learned Back Into the Planning and Tracking & Oversight Phases

All of the PIR information gained in the Evaluation phase should be collected and maintained with project information gathered during the other two phases. Developing this complete library of project information helps to establish an organizational memory in which both successes and failures can be used for learning.

There should be some mechanism or process to ensure that information is being aggregated and fed back in to improve the investment management process. For instance, the cost, risk, and benefit criteria (including the weights given to each) for the Selection phase may be refined to ensure greater implementation success of future projects. The organization should also determine whether there may be different or more appropriate cost, benefit, and risk measures that could be established that would help better monitor projects.

3.2 Evaluation Data

Data collected during the Evaluation phase will be primarily historical in nature focusing on the outcome of a project as compared to executives' expectations for the project. In addition, ex post information that is developed should include modifications made to the Selection and Control phases, as well as the institutionalized lessons-learned information. This information should be used to revise the Selection and Control phases and to help make future investment decisions.

3.2.1 Measurements of Actual Versus Projected Performance

The focus of the PIR should be on evaluating a project's actual results compared to estimates in terms of cost, schedule, performance, and mission improvement outcomes. An attempt should also be made to determine the causes of major differences between planned and end results. And the PIR should be used to help identify any inappropriate systems development and project management practices.

The PIR should provide a wide range of information regarding both the project and the process for developing and implementing the project. Specific information includes the following:

• an assessment of the project's effectiveness in meeting the original objectives,
• an identification of benefits that have been achieved,
• an assessment of whether they match projected benefits, and a determination of reasons for any discrepancies.
• an evaluation of whether original business assumptions used to justify the project were valid,
• a comparison of actual costs incurred against projected costs, a determination of how well the project met time schedules and implementation dates, management and user perspectives on the project, and
• an evaluation of issues that still require attention.

Outputs of the PIR should include user evaluations of the effectiveness of the project, actual costs broken out by category, measurements used to calculate benefits, a comparison matrix of actuals to estimates, and business-as-achieved documentation.

3.2.2 Documented "Track Record" (Project and Process)

The organization should be maintaining documentation of all decisions, changes, actions, and results that occurred throughout the project's life cycle, as well as other relevant project information, such as the business case and updated cost/benefit analyses. The organization should also be tracking recommendations (for both improving the project and refining the overall investment management process) that arise out of the PIRs.

This "track record" will be invaluable for helping the organization refine and improve its processes as more and more information is collected and aggregated.

3.3 Evaluation Decisions

A number of key decisions will be made during the Evaluation phase, including an assessment of how well the project met its intended objectives, a determination of what changes or modifications to the project are still needed, and an identification of ways to modify or improve the overall investment management process to better maximize results and minimize risks. In addition, the organization may assess the overall performance of its IM/IT investments in improving mission performance. To make these decisions, agency executives must gauge the degree to which past decisions have influenced the outcome of IM/IT projects, understand why these decisions had the effect that they did, and determine how changing the processes for making decisions could create a better outcome for current IM/IT projects and future IM/IT proposals.

3.3.1 Assess a Project's Impact on Mission Performance and Determine Future Prospects for the Project

The results and recommendations that arise out of the PIRs, combined with other project information, are a critical input for senior decision makers to use to assess the project's impact on mission performance. In making this assessment, senior managers will need to ask a number of questions about the project, including the following:

• How effective was the project in meeting the original objectives?
• Are these objectives still valid?
• Were the original business assumptions used to justify the project valid?
• What is the current status of the system?
• Are further changes or modifications necessary?

Even after a project has been implemented, decisions should be made on a regular basis about the status of the project. Senior managers should regularly question whether

1. the current system meets organizational needs,
2. the system should be modified to better meet these needs,
3. a new system is needed to best meet these needs, or
4. the needs could best be met by outsourcing the work.

In addition, because operation and maintenance (O&M) costs, for such activities as hardware upgrades, system software changes, and ongoing user training, can consume a significant amount of IM/IT resources (some have estimated that ownership costs, operations and maintenance costs, and disposition costs can consume as much as 80 percent of a project's total life-cycle costs), a plan should be developed for the continued support and operation of each IM/IT project.

3.3.2 Revising the Planning and Tracking and Oversight Phases Based on Lessons Learned

An organization's investment management process will usually not remain static, but will evolve and change over time as the organization learns more and more about what has been successful and what still needs to be improved. Modifications that may be made to the process include the following:

• changing the mixture of members on the organization's decision-making investment review group
• changing the Planning phase decision-making criteria (both screening and ranking criteria)
• changing the Tracking and Oversight phase criteria used for monitoring the progress of projects
• modifying the time frames for reviewing projects during the Tracking and Oversight phase
• modifying the triggers for identifying projects for review
• modifying the PIR methodology.

The results from one project will often not provide enough information to allow significant modification to be made to the agency's IM/IT decision-making processes. However, a significant , recurring system development problem found across multiple projects over time would because for refining or even significantly revising the organization's decision-making processes and criteria.

The causes for differences between planned and actual results should be determined and corrective actions to the overall IM/IT management process, decision criteria, etc. should be identified and documented. Once the causes for differences between planned and actual results have been determined, steps should be taken to address these causes in order to ensure greater success in the future.

All alterations or updates that are made to the Planning and Tracking and Oversight phases, based on the results of PIRs, should be documented.

• Previous Page
• Table of Contents
• Next Page