DOD Business Systems Modernization:
Navy Implementing a Number of Key Management Controls on Enterprise Resource Planning System, but Improvements Still Needed
GAO-09-841: Published: Sep 15, 2009. Publicly Released: Sep 15, 2009.
The Department of Defense (DOD) has long been challenged in effectively implementing key acquisition management controls on its thousands of business system investments. For this and other reasons, GAO has designated DOD's business systems modernization efforts as high-risk since 1995. One major business system investment is the Navy's Enterprise Resource Planning (ERP) system. Initiated in 2003, it is to standardize the Navy's business processes, such as acquisition and financial management. It is being delivered in increments, the first of which is to cost about $2.4 billion over its 20-year useful life and be fully deployed by fiscal year 2013. To date, the program has experienced about $570 million in cost overruns and a 2-year schedule delay. GAO was asked to determine whether (1) system testing is being effectively managed, (2) system changes are being effectively controlled, and (3) independent verification and validation (IV&V) activities are being effectively managed. To do this, GAO analyzed relevant program documentation, traced random samples of test defects and change requests, and interviewed cognizant officials.
The Navy has largely implemented effective controls on Navy ERP associated with system testing and change control. For example, it has established a well-defined structure for managing tests, including providing for a logical sequence of test events, adequately planning key test events, and documenting and reporting test results. In addition, it has documented, and is largely following, its change request review and approval process, which reflects key aspects of relevant guidance, such as having defined roles and responsibilities and a hierarchy of control boards. However, important aspects of test management and change control have not been fully implemented. Specifically, the program's tool for auditing defect management did not always record key data about changes made to the status of identified defects. To its credit, the program office recently took steps to address this, thereby reducing the risk of defect status errors or unauthorized changes. Also, while the program office's change review and approval procedures include important steps, such as considering the impact of a change, and program officials told GAO that cost and schedule impacts of a change are discussed at control board meetings, GAO's analysis of 60 randomly selected change requests showed no evidence that cost and schedule impacts were in fact considered. Without such key information, decision-making authorities lack an adequate basis for making informed investment decisions, which could result in cost overruns and schedule delays. The Navy has not effectively managed its IV&V activities, which are designed to obtain an unbiased position on whether product and process standards are being met. In particular, the Navy has not ensured that the IV&V contractor is independent of the products and processes that it is reviewing. Specifically, the same contractor responsible for performing IV&V of Navy ERP products (e.g., system releases) is also responsible for ensuring that system releases are delivered within cost and schedule constraints. Because performance of this system development and management role makes the contractor potentially unable to render impartial assistance to the government in performing the IV&V function, there is an inherent conflict of interest. In addition, the IV&V agent reports directly and solely to the program manager and not to program oversight officials. As GAO has previously reported, the IV&V agent should report the findings and associated risks to program oversight officials, as well as program management, in order to better ensure that the IV&V results are objective and that the officials responsible for making program investment decisions are fully informed. Furthermore, the contractor has largely not produced the range of IV&V deliverables that were contractually required between 2006 and 2008. To its credit, the program office recently began requiring the contractor to provide assessment reports, as required under the contract, as well as formal quarterly reports; the contractor delivered the results of the first planned assessment in March 2009. Notwithstanding the recent steps that the program office has taken, it nevertheless lacks an independent perspective on the program's products and management processes
- Review Pending
- Closed - implemented
- Closed - not implemented
Recommendations for Executive Action
Recommendation: To strengthen the management of NAVY ERP's change control process, the Secretary of Defense should direct the Secretary of the Navy, through the appropriate chain of command, to (1) revise the Navy ERP procedures for controlling system changes to explicitly require that a proposed change's life-cycle cost impact be estimated and considered in making change request decisions and (2) capture the cost and schedule impacts of each proposed change in the Navy ERP automated change control tracking tool.
Agency Affected: Department of Defense
Status: Closed - Implemented
Comments: Since our report, Navy Enterprise Resource Planning (ERP) has updated its Enterprise Change Request Process and Procedures to explicitly require that life cycle cost impact is estimated and considered as part of the system change control process. In addition, the change control tracking tool has been updated to capture "dollarized" cost and schedule impact, and to store supporting life-cycle cost and schedule impact information.
Recommendation: To increase the value of Navy ERP IV&V, the Secretary of Defense should direct the Secretary of the Navy, through the appropriate chain of command, to (1) stop performance of the IV&V function under the existing contract and (2) engage the services of a new IV&V agent that is independent of all Navy ERP management, development, testing, and deployment activities that it may review.
Agency Affected: Department of Defense
Status: Closed - Implemented
Comments: The Navy Enterprise Resource Planning (ERP) program office stopped the independent verification and validation (IV&V) work under the existing contract on September 30, 2009, and it awarded a new IV&V contract in September, 2010. According to the contract, the contractor must be independent of the development, testing, deployment and sustainment of the Navy ERP solution.