DOD Major Automated Information Systems:

Adherence to Best Practices Is Needed to Better Manage and Oversee Business Programs

GAO-18-326: Published: May 24, 2018. Publicly Released: May 24, 2018.

Additional Materials:

Contact:

Carol C. Harris
(202) 512-4456
harriscc@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

What GAO Found

The strength of Department of Defense's (DOD) policies for managing and overseeing major automated information system (MAIS) programs varies. Specifically, the policy for managing 24 non-business MAIS programs adheres to leading information technology (IT) management practices, but the policy for managing 10 MAIS business programs does not always do so (see table).

Analysis of Department of Defense (DOD) Policies for Managing Major Automated Information System (MAIS) Programs

 

Leading information technology management practice

Adherence to policy for non-business MAIS programs

Adherence to policy for MAIS business programs

Instituting an investment board as a process for creating and defining the membership, guiding principles, operations, roles, responsibilities, and authorities

Yes

Yes

Identifying decision authorities for making important executive-level acquisition decisions

Yes

Yes

Providing oversight whereby the organization monitors programs on their performance progress; such information includes (1) baseline estimates on cost and schedule goals and (2) thresholds to identify high risk on cost and schedule

Yes

No

Capturing and providing performance information about a particular investment (program) to decision makers at regular intervals (e.g., quarterly and annually)

Yes

No

Source: GAO analysis of agency documentation. | GAO-18-326

When DOD categorized 10 of the 34 MAIS programs as MAIS business programs, it also directed these programs to adhere to DOD's business systems policy (DOD Instruction 5000.75). However, the department directed those programs to use a policy for the management and oversight of MAIS business programs that was not fully comprehensive. Until DOD updates its business systems policy to address gaps in establishing performance information such as baseline estimates on program cost and schedule goals, identifying thresholds to identify high risk, and requiring periodic reports to be provided to stakeholders at regular intervals, stakeholders will likely not have all the information they need to manage and oversee MAIS business programs.

While all 15 business and non-business MAIS programs had either increased or decreased their planned cost estimates and the majority had delays in their planned schedule estimates, the majority of the 9 programs that had performance targets met those performance goals. Specifically, the decreases and increases in cost estimates ranged from a decrease of $1.6 billion (-41 percent) to an increase of $1.5 billion (163 percent). The decreases in planned cost were largely due to scope reduction, while cost increases were due to underestimating levels of effort and contracting issues. The slippages in schedule estimates ranged from a delay of 5 years to 5 months; these delays were caused by unrealistic expectations or unplanned changes. Six of the 9 programs that had performance targets met all of them, while the other 3 met several but not all of their performance targets. The other 6 programs were in the early stages of system development and had not begun performance testing.

Why GAO Did This Study

DOD's MAIS programs are intended to help the agency sustain its key operations. In April 2017, recognizing that MAIS programs met different mission needs, DOD categorized its MAIS programs into business and non-business systems.

The National Defense Authorization Act for Fiscal Year 2012 includes a provision for GAO to select, assess, and report on DOD's MAIS programs annually through March 2018. GAO's objectives, among others, were to (1) assess DOD's policies for managing and overseeing MAIS programs and (2) describe the extent to which selected MAIS programs have changed their planned cost and schedule estimates and met technical performance goals. To address these objectives, GAO compared DOD's policies for managing and overseeing all 34 MAIS programs (24 non-business programs and 10 business programs) to leading IT management practices. GAO also compared 15 selected programs' initial cost, schedule, and performance baselines to their current acquisition program estimates.

What GAO Recommends

GAO is making three recommendations, including that DOD update its policy for managing MAIS business programs to include baseline estimates. DOD partially concurred with this recommendation, and fully concurred with the other two recommendations. GAO continues to believe that all the recommendations are warranted.

For more information, contact Carol C. Harris at (202) 512-4456 or harriscc@gao.gov.

Recommendations for Executive Action

  1. Status: Open

    Comments: In January 2020, the Under Secretary of Defense for Acquisition and Sustainment issued an updated instruction on defense business systems requirements and acquisition, which included guidance on establishing baseline cost and schedule estimates and considering progress against the baselines at key decision points. However, the instruction does not make a distinction between initial and current baselines. Further, it did not include thresholds for cost and schedule variances or specify periodic reporting of program performance information to stakeholders. According to an official in the office of the Under Secretary of Defense for Acquisition and Sustainment, the office does not intend to add the elements of the recommendation related to thresholds and reporting. Specifically, according to the official, the office considers specifying predetermined threshold cost and schedule estimates and frequency for status reporting to be matters for implementation guidance issued by department components or determined by a program decision authority. However, until the department demonstrates that it has fully addressed the recommendation, it is limited in its ability to ensure that effective system acquisition management controls are implemented for each major business system investment and that stakeholders have the information needed to make informed decisions for managing and overseeing these investments. We will continue to monitor the department's implementation of the recommendation.

    Recommendation: The Secretary of Defense should direct the Under Secretary of Defense for Acquisition and Sustainment to update the policy or guidance for MAIS business programs. Specifically, the update should include the following elements: (1) establishment of initial and current baseline cost and schedule estimates, (2) predetermined threshold cost and schedule estimates to identify the point when programs may be at high risk, and (3) quarterly and annual reports on the performance of programs to stakeholders. (Recommendation 1)

    Agency Affected: Department of Defense

  2. Status: Open

    Comments: As of November 2019, the Department of Defense had made progress addressing the intent of the recommendation related to requirements management; however, it needs to do more to improve DHMSM program risk management. Specifically, in March 2019, the DHMSM program manager approved a requirements management plan, which includes identifying and documenting changes that should be made to plans and work products resulting from changes to the baseline requirements. Specifically, it includes forward and backward configuration and change management of the baselined requirements and managing traceability of requirements to design artifacts, test cases, defects, and change requests. However, the program has not demonstrated that it quantifies costs and benefits of risk mitigation in its risk mitigation plans. Specifically, it did not demonstrate that it had updated its guidance to require that costs and benefits of risk mitigation plans be included in these plans. We will continue to monitor the department's efforts to implement the recommendation.

    Recommendation: The Secretary of Defense should direct the Director of the Defense Health Agency to direct the program manager for the Defense Healthcare Management System Modernization program to: (1) finalize and approve its requirements management plan, (2) identify and document changes that should be made to plans and work products resulting from changes to the requirements baseline, and (3) quantify costs and benefits of risk mitigation within its program-level risk mitigation plans. (Recommendation 2)

    Agency Affected: Department of Defense

  3. Status: Closed - Implemented

    Comments: The Department of Defense agreed with our recommendation. In a March 29, 2018 memo, the department reported that the Secretary of the Navy plans to direct the program manager to identify and document all potential external environmental issues in its plan. As of August, 2019, the department has addressed GAO's recommendation. The program indicated that within the CANES Interactive Electronic Technical Manual (IETM), the CANES system has a disaster recovery plan. Based on GAO's recommendation, the program office has updated the IETM to state that CANES inherits environmental effects risks and mitigations from the platform and that CANES should be considered in all platform disaster recovery plans. The updated IETM was completed on December 30, 2018.

    Recommendation: The Secretary of Defense should direct the Secretary of the Navy to direct the program manager for the Navy Consolidated Afloat Networks and Enterprise Services program to identify and document, in the failover/recovery plan, all potential external environmental issues, such as hazards, threats, and vulnerabilities that could negatively affect work efforts. (Recommendation 3)

    Agency Affected: Department of Defense

 

Explore the full database of GAO's Open Recommendations »

Aug 3, 2020

Jun 1, 2020

Mar 5, 2020

Feb 20, 2020

  • it icon, source: PhotoDisc

    Science & Tech Spotlight:

    Deepfakes
    GAO-20-379SP: Published: Feb 20, 2020. Publicly Released: Feb 20, 2020.

Dec 12, 2019

Dec 11, 2019

Oct 30, 2019

Sep 10, 2019

Jun 26, 2019

Looking for more? Browse all our products here