ARCHIVED - Horizontal Internal Audit of Large Departments and Agencies: Contracting Information Systems and Monitoring

This page has been archived.

Horizontal Internal Audit of
Large Departments and Agencies

Contracting Information Systems and Monitoring

May 2009

Internal Audit Sector
Office of the Comptroller General

Table of Contents

Introduction

Overall Assessment

Statement of Assurance

Main Findings

Audit Objectives and Scope

Audit Approach

Detailed Findings and Recommendations

Annex A:  Participating Departments and Agencies

Annex B:  Audit Criteria and Sources

Introduction

The Treasury Board Policy on Internal Audit, which came into effect April 2006, requires the Comptroller General to lead horizontal audits in large departments and agencies (LDAs).  Horizontal audits are designed to address risks that transcend individual departments in order to report on the state of governance, controls and risk management across the federal government. This report presents the results of the horizontal audit of Contracting Information Systems and Monitoring.     

The principal authoritative reference governing Contracting is the Treasury Board Policy on Contracting, as well as key government policies and frameworks including the Management Accountability Framework, the Policy on Responsibilities and Organization for Comptrollership, the Policy on Active Monitoring, and the Policy on Risk Management. 

The objectives of the audit were to provide reasonable assurance that management receives reliable and relevant information on contracting to support informed decision making, and risk management, and effective disclosure, and that governance structures are in place to review and act upon contracting trends and risks. The scope of this engagement included contracting information systems and monitoring mechanisms that were in place between January 2008 and September 2008 in nine LDAs.  

Why It’s Important

Contracting and procurement are used extensively by all LDAs to acquire the goods and services needed to help achieve their goals and objectives, and to deliver their mandates.  Contracting and procurement have traditionally been higher risk activity areas, given the complexity of the compliance environment and rules and regulations established by the Central Agencies.  More recently, government-wide contracting and procurement practices have come under increased public scrutiny.  Canadians and Central Agencies expect that federal organizations have appropriate management systems and practices in place to monitor contracting compliance, performance, and risks on an ongoing basis.  It is expected that these management systems protect the integrity of contracting information, as well as provide senior management with reliable and relevant information to make informed decisions regarding the management of contracting and procurement activities within their departments.

In order to meet these expectations and meet the accountability requirements for contracting, it has become increasingly important to obtain relevant, reliable, and timely contracting information for decision-making.  It is essential that senior management in federal departments and agencies have the right contracting information to support their oversight roles, long term planning decisions and risk mitigation strategies with respect to the contracting function at various levels within the department.

Overall Assessment

Governance and management practices of contracting information systems and the monitoring of these activities were the focus of the audit engagement. In plain language, the audit team sought to respond to the question:    

“Does management have access to adequate information and do they have measures in place to monitor and act upon trends and emerging risks regarding contracting?”

While LDAs are beginning to make strides to have the appropriate structure and measures in place to act upon trends and risks regarding contracting, the audit identified an opportunity to move contracting information management from a transactional focus to a more strategic and integrated focus.  A more strategic approach that focuses on risks and trends at the entity level would support monitoring and risk management activities and allow departments to respond to changes and priorities more effectively.

Overall, the audit found that information, for the most part, is adequately captured and reliable systems and processes are in place to help ensure data is accurate.  However, the contracting data is not being used to the fullest extent to produce the required reporting and information for management decision-making.  The contracting data is used to support monitoring activities; however most of the monitoring is directed at the contract, project or branch level or related to Central Agency requirements such as proactive disclosure.  The majority of LDAs examined are not showing evidence of active monitoring at the entity level to identify department-wide risks and trends.  For instance, trends analysis could provide relevant information to identify potential savings or changing risk levels.

The focus of most LDAs is on transactional based activity including ensuring that individual contracts approval and management processes are in place and are operating effectively.  This transactional focus is evident from the operating level up to senior management level.  For example, the focus and mandate of many Contract Review Committees is to approve specific large contracts or monitor specific risks on a single contract. 

There in an opportunity to advance to a strategic based model of monitoring and risk management that would improve management effectiveness and apply the principles of active monitoring with regards to contracting activities.

Innovative and good practices were identified during the audit.  Documentation of contracting processes which included the identification of key control activities supporting data reliability practices was observed.   Several LDAs have established a contract management committee or review board to deal with difficult issues, help to ensure contracting instruments are being properly utilized and provide general oversight of contracting to assist in risk mitigation.  The other departments examined are in the process of establishing a contract oversight review committee.    

Statement of Assurance

In my professional judgment as Executive Director, Systems, Forensic and Horizontal Audits, sufficient and appropriate audit procedures have been conducted and evidence gathered to support the accuracy of the opinion provided and contained in this report. The opinion is based on a comparison of the conditions, as they existed at the time, against pre-established audit criteria that were agreed on with management. The opinion is applicable only to the nine LDAs examined. The evidence was gathered in compliance with the Internal Auditing Standards for the Government of Canada and the International Standards for the Professional Practice of Internal Auditing.  

 

Main Findings

Reliable Information:  Information, for the most part, is adequately captured and reliable systems and processes are in place to help ensure data is accurate.  Contracting processes are documented for the majority of LDAs, while the remainder have informal procedures documented for certain portions of the process.  Also, automated controls are in place to mitigate data integrity risks.  Improvement opportunities exist related to the documentation of data integrity risks and key controls.  Further, roles and responsibilities related to the capturing, processing, monitoring and testing of contracting data need to be clearly defined. 

Relevant Information Requirements:  All LDAs have identified information required to meet ad hoc requirements related to contracting activities.  The vast majority of the LDAs have not identified contracting information needs to support risk management and decision making at the entity level.  As a result, the information required to monitor and act upon trends and emerging risks regarding contracting is not readily available.  Improvement opportunities exist to clearly identify the information requirements and fully exploit the capability of existing information systems to ensure that the right contracting information is available in a timely manner which can be relied upon to support day-to-day monitoring practices, oversight activities, and long term planning processes. 

Monitoring:  The majority of LDAs have implemented monitoring mechanisms.  However, the focus is generally at the transactional, project or branch level to assess compliance with relevant policies and regulations.  The LDAs do not have a formal risk-based approach to contract monitoring.  There in an opportunity to advance to a strategic based model of monitoring and risk management to improve management effectiveness and apply the principles of active monitoring with regards to contracting activities.

Reporting:  Reporting is generally ad hoc and transactional in nature.  Other than Central Agency reporting requirements, there are little or no other systematic reporting generated to provide information in support of decision making and risk management processes.  There is an opportunity to implement strategic reporting processes to support LDAs in their assessment of performance, in their ability to address trends and deficiencies in management practices and controls, and to enable better communication to effect change. 

Audit Objectives and Scope

The objectives of the audit were to provide reasonable assurance that:

• management receives reliable and relevant information on contracting to support informed decision making, risk management, and effective disclosure; and
• governance structures are in place to review and act upon contracting trends and risks.

The scope of this engagement included contracting information systems and monitoring processes that were in place between January 2008 and September 2008 in nine LDAs.   The engagement did not assess grants and contribution arrangements nor interdepartmental arrangements.

Audit Approach

The work completed under this audit included various tests, as considered necessary, to provide reasonable assurance that audit objectives were achieved.  Consultants were engaged to support the Office of the Comptroller General audit team in the conduct of this horizontal audit engagement.  The consultants assisted in the planning, conduct and reporting phase of the audit.       

The audit engagement work was completed in two phases.

Phase 1:  This phase consisted of a preliminary survey, identification of risks and key audit criteria, and the development of the audit program.  During this phase, contracting data was gathered from all LDAs for a risk assessment to assist in the selection of organizations to include in the audit engagement. 

A questionnaire was sent and responses received from all LDAs.  The questionnaire focused on contracting risks, controls, reporting, monitoring and governance.  Documents were gathered to support the responses.  Based on the contracting data, documents provided and analysis of the responses received, departments and agencies were selected for the audit based upon higher risk.  Specifically, organizations were selected based a spectrum of organizational complexity, procurement and contracting complexity, degree of formalized processes and proportion of service contracts.  Based on these selection criteria, nine LDAs were selected for the conduct phase of the audit.  See ANNEX A for a list of LDAs included in this audit.

As part of the audit program, audit criteria were established to measure practices against established and accepted sound management practices.  Audit tools and testing procedures were developed and applied for each of the criteria for this audit and included the gathering of evidence and analysis related to the criteria listed in ANNEX B. 

Phase 2:  The conduct and reporting phases consisted of the execution of the audit program.  Audit procedures were used including interviews with key stakeholders, documentation review and analysis.  Documentation and interview notes were analyzed and compared to key controls for each criterion for the purpose of assessing results and findings.  The nine LDAs were then assessed as to whether they fully met the audit criteria, met them with some exceptions, or did not meet them.  This provided a basis for reporting results horizontally and identifying trends and patterns. 

Management Action Plans:  The findings and recommendations of this audit were presented to each department included in the scope of the audit.  These departments have reviewed the recommendations, provided responses and developed management action plans as they deemed necessary.  The Departmental Audit Committees of those departments will be briefed on the audit findings and the departmental responses.  They may also be requested by the deputy head to recommend for approval the management action plans.  The Departmental Audit Committees will periodically receive reports from management on the actions taken where management action plans are in place.

Deputy heads of other large departments will take into account the results of this horizontal internal audit and will ensure that management action plans are developed where they deem necessary.  The Departmental Audit Committee of these departments will be briefed regarding this audit.

Detailed Findings and Recommendations

Finding 1:  Reliable Information

Reliable information is necessary to support sound planning, decision-making and reporting.  By implementing appropriate controls over data accuracy and completeness, and ensuring such controls are documented, applied and tested, management can derive further assurance of the reliability of the information received.

The audit team examined the key controls supporting data accuracy and completeness, including the establishment of formal contracting processes, the documentation of risks and key controls, and the application of automated controls related to contracting information systems.  The audit team observed that contracting processes have been documented for the majority of the LDAs while the remainder have informal procedures documented for certain portions of the process.

The majority of the LDAs examined have various processes in place to assess and mitigate risks related to data accuracy.  These processes range from the establishment of a joint committee between IT and Finance to identify and control data risks at a strategic level to the identification of data weaknesses at the operational level.  A number of LDAs reviewed, on a sample basis, transactional data and reconciliations to identify potential inaccuracies.  None of the LDAs specifically identified nor documented risks and key controls directly related to their contracting processes.  As a result, while the processes to assess and mitigate data accuracy risks may help identify some issues and improvement opportunities, identifying and documenting significant risks and key controls would enhance the ability of management to ensure that appropriate controls are in place and that they are functioning as intended.  Also, the audit found that roles and responsibilities for data integrity related to contracting activities have not generally been formally assigned.

The vast majority of the LDAs examined have contracting systems that employ the use of automated controls to reduce the risk of inaccurate data input.  As well, several other good practices have been observed including: the documentation of contracting processes and practices in flowchart format which includes the identification of control activities and key decision points; the use of on-line contracting and procurement tools; as well as the use of system exception reports to identify inconsistent information between systems.

Recommendations:

Finding 2:  Relevant Information Requirements

In order to ensure that relevant information is available to meet the needs of managers for monitoring and reporting on contracting activities, the audit team expected that information needs would be identified.  In order to identify information needs, we expected to find that LDAs would have considered the goals and objectives of the function, as well as the reporting and monitoring requirements. 

The audit team found that, other than Central Agency reporting requirements for quarterly disclosure and year-end reporting, and ad hoc report requests focusing at the transactional level, information requirements have not been formally identified.  Some LDAs have identified a list of planned or “wish list” information needs on an ad hoc manner.  Very few of the LDAs have identified or defined the contracting information required to support risk management and decision making related to contracting activities.  For those who have, the focus was at the transactional level.

Contracting information to support risk management and decision making at the entity level could include the following:

• contracting data and trends to understand the types of services being outsourced;
• extent and use of standing offers;
• instances of not using a standing offer where such a vehicle should have been used;
• proportion of sole sourced contracts under $25K;
• amendment analysis by vendor;
• contracting in excess of $25K that was sole sourced; and
• number of contracts and value of contracting over a period of time by vendor.

It is essential that management in federal departments and agencies have the right contracting information on a timely basis. This information will support their monitoring practices, oversight activities, day-to-day and long term planning decisions.  It will also enhance their ability to support their risk mitigation strategies with respect to the contracting function. 

Recommendation:

Finding 3:  Monitoring

The TB Policy on Active Monitoring requires that departments establish a capacity to actively monitor, on an ongoing basis, management practices and controls by developing and maintaining an ability to detect and communicate within the organization, as early as possible, significant risks, potential and actual control failures, and other significant management vulnerabilities.  The audit team expected to find appropriate mechanisms for monitoring contracting risks at all levels, including the risks of compliance with applicable policies and regulations.        

The audit team observed that the majority of the LDAs have implemented some form of monitoring mechanisms to assess compliance with contracting policies and regulations at the transactional level.  These monitoring mechanisms include senior management meetings and various oversight committees that meet weekly and/or monthly.  These mechanisms generally include monitoring of individual contracts, specific procurement vehicles used, contracting spend status, and potential non-compliance issues. 

In general, monitoring requirements related to contracting risks are informally identified and documented and/or considered through the transactional and procedural activities that are performed.  Further, key compliance risks are monitored more on a detective basis and this monitoring tends to be more financially focused.  For instance, monitoring of compliance risks includes post-payment verification activities and/or invoice-based monitoring to identify billing discrepancies and trends.  The audit team expected that LDAs would have a formal, documented process in place to identify, assess, and update key contracting risks at all levels within the organization.  Such an approach could provide a departmental perspective of the risks related to contracting activities.  For instance, contracting trends analysis within a department could provide information about potential risks surrounding contract splitting and employer/employee relationships. The audit team found that LDAs audited do not have a formal risk-based approach for contract monitoring.

Furthermore, the audit considered how LDAs are incorporating compliance with the Policy on Green Procurement in contract monitoring activities.  While many LDAs have identified Green Procurement as a consideration in their contracting process, monitoring with respect to the compliance with the Green Procurement policy was observed in only one LDA.

It should be noted that some of the LDAs are in the process of developing monitoring indicators and performance measures at the entity level in support a risk-based monitoring approach.  A risk-based approach to monitor contracting activities at the entity level can help organisations identify and manage risks in a more effective and efficient manner.  Such an approach increases the organization’s ability to evaluate overall contracting performance, analyze trends, and amalgamate contracts across the departments.  This may result in achieving opportunities for cost savings, as well as reducing contracting risks such as contract splitting, sole source challenges, and make/buy decisions.  

Recommendation:

Finding 4:  Reporting

Effective reporting enables timely and effective action to assess performance, address trends and deficiencies in management practices and controls and enable better communication to effect change.  The expectation was that departments had established effective reporting systems, including standard contract reporting generated regularly and in a timely manner focussing at the departmental, operational and transactional level.

The audit team observed that other than Central Agency reporting requirements for quarterly disclosure and year-end reporting, there were little or no other systematic reporting generated to provide information for management decision making and contract related risk management at the entity level.  The majority of the LDAs produce contracting reports at the transactional level or branch level and related compliance with the Financial Administration Act.  Reports are periodic, ad hoc and reactive in nature.  Further, none of the LDAs have implemented a formalized process to assess the continued relevance of the contract reporting that was being completed.

It was also noted that contract reporting generally requires significant manual intervention.  Few LDAs generate reports directly from their Enterprise Resource Planning (ERP) system.  The process typically begins with a data extract from the ERP system followed by use of a spreadsheet to manipulate and present the information requested.  In the majority of LDAs, there is a general belief that the systems are in place to meet some reporting requirements; however, the full capability of the systems is not fully exploited.

The absence of appropriate and timely financial and non-financial reporting related to contracting activities does not support an effective governance structure that is required to ensure that management identify, analyze and act on trends and issues. 

Recommendation:

Annex A:  Participating Departments and Agencies

Based on the applicable selection criteria, nine LDAs were selected for the Examination Phase of the audit as follows:

• Canada School of Public Service (CSPS);
• Canadian Space Agency (CSA);
• Citizenship and Immigration Canada (CIC);
• Department of National Defence (DND);
• Department of Finance Canada (Finance Canada);
• Immigration and Refugee Board (IRB);
• Indian and Northern Affairs Canada (INAC);
• Natural Resources Canada (NRCan); and
• Public Works and Government Services Canada (PWGSC).

Annex B:  Audit Criteria and Sources

Contracting and Information Systems Audit Criteria and Sources

Criteria	Source
1. Appropriate key controls over data accuracy are documented, applied and tested.	TBS Core Management Controls, Policy on Risk Management and Policy on Contracting
2. Reporting of information to support informed decision making, risk management and disclosure is prepared and communicated.	TBS Core Management Controls, TBS Management Accountability Framework, Policy on Active Monitoring and Policy on Risk Management
3. Monitoring and reporting to manage risks and help ensure contracting complies with central agency and department/agency requirements.	TBS Core Management Controls, Policy on Active Monitoring and Policy on Risk Management
4. An effective governance structure exists to identify, challenge and monitor contracting activity and resulting management actions.	TBS Core Management Controls, TBS Management Accountability Framework and Policy on Risk Management