ARCHIVED - Internal Audit and Evaluation Bureau - Audit of Electronic Record Keeping

• Previous Page
• Table of Contents
• Next Page

This page has been archived.

3.0 Audit Results     

Each area of focus was assessed against audit objectives and related audit criteria. The audit results are presented below by line of enquiry.

3.1 Policy and Governance

As part of the audit, we examined the Secretariat's policy and governance capacity relative to electronic record keeping. We expected to find that governance structures were in place to effectively support an IM strategy and IM outcomes, in particular electronic record keeping. Specifically, we expected that:

• Governance structures, mechanisms, and resources were in place to ensure continuous and effective management of information; and
• Monitoring and reporting processes were in place.

Overall, we found that a governance framework was in place that defined IM roles and responsibilities. While monitoring and reporting of IM activities were occurring, improvements could be made to ensure effective management. 

We found that an IM governance structure was in place to support electronic record keeping within the Secretariat. The Secretariat's Management and Infrastructure Committee, comprising senior management at the Assistant Secretary level, has a subcommittee dedicated to IM and IT. Responsibility for overseeing IM within the Secretariat is assigned to the EIMS group within IMTD.

Roles and responsibilities for IM were defined and communicated as follows:

• Organization and committee charts which demonstrate reporting relationships;
• Committee terms of reference which defined the scope of committee activities and included oversight of IM activities;  and
• Departmental guidelines and policies which reflect these roles and responsibilities. and are updated as necessary.

In line with the periodic review requirements of the Policy on Information Management, the Secretariat was in the process of updating its IM strategy at the time of our audit. High-level IM strategies were presented and approved through the governance structure by the Secretariat's  Management and Infrastructure Committee. This update included a review of the operating environment and consultation with senior internal stakeholders. In addition, a plan to support the implementation of these strategies was under development at the time of our audit. We noted that electronic record keeping was treated not as a discrete activity but as part of the broader need for managing information. The current strategy and plan were articulated when IMTD was part of the Department of Finance Canada in 2008–09. We noted that the timing and the frequency of review complies with the Policy on Information Management.

In reviewing the IM planning process, we were told that planning is largely based on historical activities rather than on forecasted needs. In addition, many IM services, including activities related to electronic record keeping, were delivered in response to ad hoc requests from sectors rather than planned activities arising from a formal mechanism that identifies IM needs from sectors in advance.     

A related issue is the absence of defined measures of performance. While Departmental Performance Reports report on IM activities, there are no defined performance expectations and measures for IM strategies or activities. For example, EIMS carries out IM training activities, but the intended outcomes have not been defined nor have performance measures been identified. With defined desired outcomes and related performance measures, the Secretariat would be better positioned to determine whether intended outcomes were being achieved.

In setting performance expectations, the department should distinguish between performance measures for ongoing activities and those for project-driven activities with end dates (such as those associated with activities undertaken to implement a strategy). Ongoing activities require measures that indicate success and are largely aimed at defining satisfactory performance on an ongoing basis. Examples may include numerical targets and quotas, such as records created, user statistics, and error rates. While this is also true for project-driven activities, clear, phased deliverables should also be defined, with anticipated completion dates that reflect the sequential nature of project life cycles. This would affect many of the activities that involve the creation, development, and implementation of processes.

We found that monitoring of IM activities was done as part of regular departmental reporting mechanisms, but performance information largely pertained to activities undertaken during the reporting period and focused on outputs without reporting on outcomes. Also, we were informed that there was no formal monitoring of compliance with IM practices and therefore no reporting on monitoring. 

Without clear performance expectations and performance measures, the Secretariat may find it difficult to measure either its progress toward or its success in achieving intended objectives.

In addition, without some form of monitoring, the Secretariat cannot determine whether its practices comply with its policies or whether these policies are sufficient. 

Recommendation 1

The CIO should define outcome performance expectations and performance measures for IM strategies and operational activities and should ensure periodic reviews and reports on performance results (including compliance) against these expectations. 

Recommendation 2

The CIO should define monitoring and reporting roles and responsibilities for IM in order to meet the needs of the Secretariat and to ensure that IM strategies and goals are met. This should be done by leveraging the knowledge of sectors and defining their responsibilities for IM, while respecting the holistic IM stewardship responsibilities of the EIMS group within the Secretariat. Once defined, these roles and responsibilities should be approved by the Secretariat's governance committees to ensure acceptance.

3.2 People and Capacity

We examined how the Secretariat developed people and capacity to support its electronic records management activities. We expected to find that the Secretariat was developing highly skilled workforces to ensure that capacity exists to deliver IM outcomes. Specifically, we expected that the Secretariat:

• Had a common body of knowledge, learning and assessment tools; and
• Had a common understanding of common policy instruments and assessment tools.

We found that the Secretariat used various processes to support the development of highly skilled workforces that supported sound electronic IM practices. However, available learning resources have not been universally leveraged throughout the Secretariat, and opportunities exist to improve planning for these activities.

As previously stated, EIMS is the group within IMTD that is responsible for overseeing IM activities, including those pertaining to electronic records management, within the Secretariat. Its role in IM service delivery includes internal development of policy instruments, training and development, and promotion of IM in order to build awareness of the importance of IM. EIMS informed us that they are reorganizing, and transitioning from an organization of specialists in paper-based records management to one that is better able to support electronic record keeping and overall IM stewardship functions. 

In reviewing EIMS activities, we assessed the delivery of its services by interviewing staff and reviewing IMTD performance reports, training statistics, and training schedules that were available at the time of our review. We noted that IM practitioners regularly take IM courses to maintain and expand their knowledge of IM practices as required under the Policy on Information Management.

There was a uniform understanding of the importance of IM across the Secretariat. We found that standardized training formats and general IM policy/guidance documents were in place to support Secretariat staff's understanding of IM. Furthermore, the Secretariat's orientation courses include brief discussions of IM concepts, and all staff members receive a security briefing upon arrival that includes elements of IM. Although training was available, there was no mandatory IM and electronic record-keeping training. As a result, IM understanding varies across the Secretariat, since staff members are not receiving consistent and mandatory training. 

The issue concerning the development of skilled workforces is discussed further in section 3.5 of this report and in the associated recommendation.

3.3    Enterprise and Information Architecture

As part of the audit, we examined the manner in which the Secretariat has been developing enterprise and information architecture. We expected to find that the Secretariat was developing information architecture and processes that respected their IM risks and controls, and operational requirements. Specifically, we expected that:

• Information and records would be identified and managed as valuable assets to support the outcomes of programs and services, as well as operational needs and accountabilities; and
• The Secretariat's programs and services would provide convenient access to relevant, reliable, comprehensive and timely information.

For enterprise and information architecture, we found that the Secretariat has not been consistently developing information architecture and processes aligned with their IM risks and controls, and operational requirements. Practices vary by sector and user group.

At the corporate level, we found that the Secretariat has periodically performed corporate risk assessments and has established a Corporate Risk Profile. The risk areas identified included IM. The Secretariat' risk methodology included an assessment of impact and likelihood, as well as the development of mitigation strategies to ensure that risk areas are addressed. However, our review found that mitigation strategies to address identified IM risk areas tended to be high-level and long-term, which might expose the Secretariat to identified risks in the short-term.

We were informed that sectors did not regularly analyze business processes with a goal of identifying IM needs. Instead, sectors employ tools (e.g., electronic systems) available within the Secretariat to manage information and build their processes to these systems. Staff indicated that only when a need for a new electronic system was identified would they analyze their processes and consult with EIMS staff.

EIMS staff works with client sector staff to develop tools and IM processes that support the sector's business activities. In particular, we were told that this primarily results in the development of file plans. Although we confirmed that all sectors had established file plans, sector staff expressed differing views on the accuracy of these plans.

There is no set schedule for the review of sector file plans, but it is incumbent on client groups to identify issues and bring them to the attention of EIMS. A regular review process would help ensure that file plans remain accurate and support users' needs. 

Although we found that general guidelines are available to support the development of naming conventions and that sharing of practices and consultations across sectors are occurring, sectors are at different stages of implementing independent IM practices. 

We found that various generic documents exist within the Secretariat to help with the identification of information of business value and choice of repository. However, few documents had been developed to provide details on specific record-keeping practices at the Secretariat level. Rather, they were at the sector or division level. In many cases, this often resulted from sectors identifying a need for expanded guidance on IM practices and the assignment of IM roles and responsibilities for this task within their sector through sector IM working groups or IM champions.

Nonetheless, the Secretariat staff expressed differing views about the consistency as well as the clarity of IM practices across the Secretariat. Department-wide tools and applications are further discussed in section 3.4 of this report. 

3.4    Information Management Tools and Applications

We examined the extent to which the Secretariat developed and implemented IM tools and applications to support its electronic record-keeping practices. We expected to find that IM tools were developed and implemented that respect appropriate control requirements of the Department and of the business users. Specifically, we expected that the Secretariat had developed and implemented common and enterprise-wide tools and applications.

Overall, we found that the extent to which key methodologies, mechanisms, and tools were established and implemented to support departmental record keeping throughout the Secretariat varied by sector, with few consistent practices across the department.

Within the Secretariat, the main repository for unstructured corporate electronic information is the Records, Document and Information Management System (RDIMS), which was the government-wide solution for electronic record keeping, and implemented in the Secretariat in 2000. However, Secretariat staff has other options for information storage. In addition to RDIMS, repositories include shared drives, personal drives, Microsoft Outlook, paper files and the Corporate Information Centre (CIC), which is a centralized records office in the Secretariat.

Guidance documents related to IM and records management indicated a clear preference within the Secretariat for storage in RDIMS. In addition, staff members are aware of this preference. However, RDIMS use is not mandatory department-wide. 

EIMS estimates that a significant portion of unstructured information is saved in repositories other than RDIMS, which was confirmed through our audit testing. 

We identified several possibilities for the low adoption rate of RDIMS, including:

• User acceptance: Proficiency and comfort with the system were identified by staff as possible barriers to its use because staff were not comfortable searching the repository for information and/or were unfamiliar with the system to accurately save the corporate information;
• Reliability: It was expressed by multiple staff members that RDIMS was not user friendly, was antiquated, and did not meet their needs, or a combination of these;
• Understanding of the aspects of IM: General policies and guidelines define business value as information records that have enduring value. However, in practice, staff members are confused about the difference between transitional and corporate information;
• Alternative options: Because of the availability of other repositories, users may have deliberately chosen to use these other preferred repositories;
• System limitations: Incompatibility of RDIMS with some software in use within the Secretariat means that users are unable to store information in the repository;
• Network limitations: The security certification of the network does not support storage of information above a certain security classification.

Although anecdotal evidence suggests duplication of information across repositories is occurring, the extent to which these information resources (outside RDIMS) have a corporate value is unclear. EIMS has indicated that other than conducting a manual review, it cannot estimate the extent of duplication based on the existing tools. The risks arising from the potential duplication may need to be considered and managed. These risks include:

• Accessibility of information:  If information is duplicated across repositories, there is a risk that information of business value may go missing and/or that incomplete or inaccurate information may be used in decision making. 
• Increased costs: If information is duplicated across repositories, there are costs associated with acquiring additional storage and maintaining this additional storage, as well as costs associated with increased workloads to support expanded search, retrieval, and review of information for activities such as access to information requests.

As previously stated, staff expressed differing views about the accuracy of their file plans and the extent to which they support their business needs. The RDIMS credibility issues identified above may influence this view.  

While the establishment of file plans and naming conventions is a positive step, the absence of department-wide practices in this area may create barriers to information sharing across the Secretariat, as staff in different sectors may be unfamiliar with how to store or locate information in another sector and therefore may misfile or be unable to locate required information. Furthermore, it may also encourage additional information duplication, as users store additional copies or versions of information already in the corporate repository or elsewhere for their individual use or group use.

In terms of responsibilities for the various repositories:

• EIMS is responsible for the overall maintenance and support of all repositories and for the IM life-cycle management of RDIMS records and records retained centrally by CIC;
• Sectors, directorates and individuals, or a combination of these, are responsible for maintaining their records retained in repositories outside RDIMS, such as shared drives, personal drives, Outlook and paper files. However, EIMS retains overall responsibility for these repositories.

Limited guidance was found on the differences in use of the various repositories, and we found that formal monitoring or reporting on compliance with existing guidance was not occurring. 

Although we did not find common enterprise-wide IM practices in place within the Secretariat during our review, we noted some examples of strong or leading IM practices. These are presented in Appendix C.

Recommendation 3

The CIO should develop an inventory of existing IM practices and should identify key practices that may be transferable or applicable to the Secretariat as a whole. EIMS should develop department-wide IM practices and tools based on these key practices, as appropriate, and should ensure that existing sector capabilities are leveraged to support their development and implementation.

3.5 Information Management and Service Delivery

Finally, we examined the manner in which the Secretariat delivered IM and services in support of its operations. We expected that record-keeping practices would ensure the provision of timely, accurate, and accessible information, in support of the delivery of the Secretariat's programs and services. Specifically, we expected that:

• All information would be managed to ensure the relevance, authenticity, quality, and cost-effectiveness of the information for as long as it is required to meet operational needs and accountabilities; and
• The Secretariat programs and services would integrate IM requirements into development, implementation and reporting activities.

Overall, we found that record-keeping practices (especially electronic) were not consistently in place to ensure timely, accurate and accessible information. However, EIMS is aware of IM practice weaknesses and is working on implementing its revised IM strategy to improve electronic records management.

As previously discussed, we were told that EIMS staff work with sectors to develop a file plan (including electronic) to support sector activities. However, our testing found that instead of being based on sector activities, file plans were based on organizational structures. From an organizational perspective, activities have more permanence than organizational structures; this may be a factor in why staff members do not believe that file plans support their work. EIMS indicated that retention periods have been set for all Secretariat information resources. Our audit testing found that all sectors included in this audit had retention schedules, most of which had been set, with the remaining under review. EIMS staff also told us that the Secretariat applies the Retention Disposition Authorities (RDA) or the Multi-institutional Disposition Authorities (MIDA) or the Institutional Specific Disposition Authorities (ISDA), and that retention periods for most information resources are established. 

We found that a disposition process has been developed and implemented for records retained centrally by CIC (i.e., paper records). The extent to which a designed and implemented disposition process was in place depends on the repository in which the information is held, as well as the sector, directorate, division, and user. RDIMS is a central repository; however, it was found that a disposition process has not yet been defined and implemented owing to system limitations. Within the Secretariat, users also have the ability to store information in a variety of repositories (mainly electronic). Our audit interviews found that while some sectors had a defined process to transfer or dispose of some types of information in these other repositories, the overall consensus was that a universally defined disposition process was not in place within the Secretariat. However, at the time of our audit, work was underway by EIMS to develop a risk-based disposition process for unstructured electronic records.

We reviewed RDIMS documentation to determine the extent to which electronic information was being retained longer than necessary in sectors included in the scope of our audit. Our testing found that the majority of information in RDIMS has not yet exceeded its retention periods. However, it is important to note that RDIMS was implemented in the Secretariat in the year 2000 and that a large majority of retention periods set for the various folders of sectors included in the scope of our audit have been set for a duration longer than the time which has elapsed since RDIMS' implementation. Therefore, if no disposition process is implemented for unstructured electronic data, this finding will likely significantly change and pose risks in the short-term. 

Inconsistent disposition processes, coupled with the potential duplication previously mentioned, may lead the Secretariat to dispose of one version of a record, but other copies may be retained. This risk will increase if formal disposition processes for the repositories are not consistently rolled out, since the repositories often work in coordination. This could further complicate information storage and retrieval.

From a departmental standpoint, we found that IM requirements are addressed during departmental strategic planning. In setting the revised IM strategic plan, we found that the Secretariat carried out a review of the operating environment, including consultation at senior levels. This was done to guide the development of an IM strategy for the Secretariat. We note that this revised strategy recognizes some of the issues raised in this report and that some work has begun to strengthen IM practices in the Secretariat.

At the operational level, we found that IM requirements are considered by program staff at the time of system design, but not consistently in program process design/review. We also found that few sectors had retained documentation on their information requirements or were aware whether these were ever documented. As program information needs evolve, staff should consider IM impacts and required changes within existing systems and processes on an ongoing basis to meet business needs. Instead, through our interviews with staff, we found that IM considerations are only considered when building or acquiring a system. 

Much of the work of the Government of Canada is dependant on managing information, and the Secretariat is no exception to this concept. Therefore, it is crucial that information needs drive the development of business processes. While technology frequently enables innovation and processes, IM activities should be guided by current business needs. Also, if business processes are not documented, it is difficult to periodically review them for required adjustments and improvements.

Recommendation 4

The CIO, in conjunction with sectors, should perform a gap analysis of the IM life cycle of electronic records to ensure that consistent IM life-cycle practices are in place across the Secretariat and information repositories, ensuring that:

• Information needs and processes are defined, documented, and periodically reviewed for all of the Secretariat's user groups; and
• Policies and training are updated to support the consistent application of these practices and to meet the needs of the Secretariat's users

3.6 Overall Conclusion

We conclude with a high level of assurance that although key aspects of a management control framework over unstructured electronic record keeping are in place within the Secretariat, a number of significant improvements are necessary in order to fully ensure the provision of relevant, timely and accessible electronic information to support decision making and general IM practices. The Secretariat was in the process of implementing its revised IM strategy to improve electronic records management at the time of our audit. However, some further improvements are necessary. Specifically:

• Policy and Governance:  A governance framework is in place within the Secretariat that defines IM roles and responsibilities to support unstructured electronic record keeping at each level. Although key monitoring and reporting processes are in place, improvements to planning, performance measurement and compliance monitoring would further strengthen the governance framework.
• People and Capacity:  Some processes to support the development of highly skilled workforces are in place, but there is room for improvement. Specifically, available learning resources are not being universally leveraged by staff, and opportunities exist to improve planning for these activities.
• Enter prise and Information Architecture:  The Secretariat has not consistently been developing information architecture and processes that respect IM risks, controls, and operational requirements. Practices vary by sector and user group.
• Information Management Tools and Applications: The extent to which key methodologies, mechanisms, and tools have been established and implemented to support departmental record-keeping throughout the Secretariat varies by sector, with few consistent practices department-wide.
• Information Management and Service Delivery: Record-keeping practices have not been fully implemented to ensure that information is timely, accurate, and accessible.

• Previous Page
• Table of Contents
• Next Page