Skip to main content

DOD Major Automated Information Systems: Improvements Can Be Made in Applying Leading Practices for Managing Risk and Testing

GAO-17-322 Published: Mar 30, 2017. Publicly Released: Mar 30, 2017.
Jump To:
Skip to Highlights

Highlights

What GAO Found

Most of the 18 selected Department of Defense (DOD) major automated information system (MAIS) programs that GAO reviewed had experienced changes in their planned cost and schedule estimates and half of the programs had met their technical performance targets. Specifically, 16 programs experienced changes in their cost estimates ranging from a 39 percent decrease ($1.47 billion) to a 469 percent increase ($1.63 billion). The average cost increase was $457.2 million among the 11 programs reporting an increase. Fourteen programs experienced schedule delays, which ranged from 2 months to over 13 years. Finally, half of the MAIS programs fully met all of their technical performance targets. Of the remaining nine programs, 4 four had partially met their target because each was still conducting tests. The other five programs were in the early stages of system development and had not begun testing.

In addition, for the five MAIS programs GAO selected for in-depth review, all had either fully or partially applied leading practices for managing requirements, risks, and for conducting systems testing and integration.

  • Managing requirements. Three of the five programs had fully implemented the practices for managing requirements, while the other two had partially implemented some practices. Leading practices in this area include establishing requirements and ensuring traceability between requirements and work products.
  • Managing risks. Three of the five programs had fully implemented the risk management practices, while two had partially implemented some practices. An effective risk management process identifies potential problems before they occur. For example, one Army program did not have standard operating procedures for defining thresholds or bounds to manage risk. Unless such procedures are defined, the program will not have the tools needed to define risk management activities, including whether and how certain risks are prioritized. Further, programs should include practices to identify potential problems so that risk-handling activities may be planned and invoked across the project to mitigate the potential for adverse impacts. However, one Air Force program did not develop an overall risk mitigation plan to guide the implementation of individual risk mitigation activities. Without an overall risk plan to guide individual development efforts, those efforts cannot be managed cohesively.
  • Testing and integration. Four of the five programs had fully implemented practices for systems testing and integration. Programs should, among other activities, establish roles and responsibilities to manage testing and integration activities, including a chief developmental tester to oversee testing activities. However, one Air Force program reported difficulty in hiring a qualified individual to perform these duties. Until this position is filled, the program may not effectively manage risks and verify compliance with system acquisition and operational requirements.

Why GAO Did This Study

DOD's MAIS programs include systems that are intended to help the department sustain its key operations. The National Defense Authorization Act for Fiscal Year 2012 includes a provision for GAO to select, assess, and report on the department's MAIS programs annually through March 2018. This is GAO's fifth report and (1) describes the extent to which selected MAIS programs have changed their planned cost and schedule estimates and met technical performance targets and (2) assesses the extent to which selected MAIS programs have used leading IT acquisition practices, including risk management.

GAO selected and reviewed cost, schedule, and performance data for 18 of DOD's MAIS programs that were non-classified and had an acquisition performance baseline. In addition, GAO performed an in-depth review of 5 of the programs, comparing selected IT management practices used by them to leading practices for requirements and risk management and systems testing and integration. The five selected programs were from at least two military services and had not been assessed by GAO in the past year. GAO also interviewed relevant program officials.

Recommendations

GAO recommends that DOD improve the management of specific MAIS programs, including establishing procedures for defining risk thresholds, developing an overall risk mitigation plan, and filling a key test management position. DOD concurred with all of the recommendations.

Recommendations for Executive Action

Agency Affected Recommendation Status
Department of Defense To help improve the management of DOD's MAIS programs, the Secretary of Defense should direct the Secretary of the Army to direct the program manager for Global Combat Support System-Army Increment 1 to establish standard operating procedures for managing risks that include guidance for establishing thresholds and bounds for key risk areas.
Closed – Implemented
The Department of Defense concurred with and implemented our recommendation. On March 1, 2017, the project manager for the Army's GCSS-A Increment 1 program amended the Risk Management Plan that addressed the our recommendation to establish procedures for managing risks. For example, the project manager amended the Risk Management Plan on March 1, 2017. The updated plan includes revised procedures that requires the risk owners to work with the risk management officer to define the conditions for establishing threshold and bounds.
Department of Defense To help improve the management of DOD's MAIS programs, the Secretary of Defense should direct the Secretary of the Air Force to direct the program manager for Air and Space Operations Center-Weapon System Increment 10.2 to develop an overall risk mitigation plan to guide the implementation of individual risk mitigation and contingency plan activities.
Closed – Not Implemented
The Department of Defense concurred with but did not implement our recommendation. In April 2017, the department stopped all work efforts for the Air and Space Operations Center-Weapon System Increment 10.2 program. In January 2018, the Secretary of the Air Force officially cancelled the program altogether and, as a result, the risk mitigation plan was not developed.
Department of Defense To help improve the management of DOD's MAIS programs, the Secretary of Defense should direct the Secretary of the Air Force to direct the program manager for Joint Space Operations Center, Mission System Increment 2 to appoint a chief developmental tester to oversee systems testing and integration activities.
Closed – Implemented
The Department of Defense concurred with and implemented our recommendation. In July 2017, the program manager appointed a chief development tester to oversee the systems testing and integration activities for the Joint Space Operations Center, Mission System Increment 2 program.

Full Report

GAO Contacts

Topics

Best practicesCost estimatesDefense capabilitiesInformation systemsIT acquisitionsMilitary forcesPerformance measurementProgram evaluationRisk managementTestingSchedule slippagesRequirements definition