Software Development: DOD Faces Risks and Challenges in Implementing Modern Approaches and Addressing Cybersecurity Practices

GAO-21-351 Published: Jun 23, 2021. Publicly Released: Jun 23, 2021.
Jump To:
Fast Facts

The Department of Defense plans to spend $12 billion on its 29 largest business information technology systems during FYs 2019-2022. DOD's efforts to modernize business systems has been a topic on our High Risk List since 1995.

We found:

  • 22 programs that were actively developing software reported using approaches that reduced risk of cost and schedule overruns, such as early cybersecurity testing
  • DOD may be underestimating the risks for some of its acquisitions
  • DOD has taken steps to improve the sharing and transparency of data it uses to monitor acquisitions but needs to do more

We made 2 recommendations to improve acquisition monitoring.

Aerial view of the Pentagon

Skip to Highlights
Highlights

What GAO Found

According to the Department of Defense's (DOD) fiscal year (FY) 2021 budget request, DOD spent $2.8 billion on the 29 selected major business information technology (IT) programs in FY 2019. The department also reported that it planned to invest over $9.7 billion on these programs between FY 2020 and FY 2022. In addition, 20 of the 29 programs reported experiencing cost or schedule changes since January 2019. Program officials attributed cost and schedule changes to a variety of reasons, including modernization changes and requirements changes or delays. Seventeen of the 29 programs also reported experiencing challenges associated with the early impacts of the COVID-19 pandemic, including the slowdown of contractors' software development efforts.

DOD and GAO's assessments of program risk identified a range of program risk levels and indicated that some programs could be underreporting risks. Specifically, of the 22 programs that were actively using a register to manage program risks, DOD rated nine programs as low risk, 12 as medium risk, and one as high risk. In contrast, GAO rated seven as low risk, 12 as medium risk, and three as high risk. In total, GAO found 10 programs for which its numerical assessments of program risk reflected greater risk than reported by DOD, while DOD had three programs with greater reported risk than GAO. DOD officials noted that differences in risk levels might be associated with a variety of factors, including different risk assessment approaches. However, the differences in risk level GAO identified highlight the need for DOD to ensure that it is accurately reporting program risks. Until the department does so, oversight of some programs could be limited by overly optimistic risk perspectives.

As of December 2020, program officials for the 22 major DOD business IT programs that were actively developing software reported using approaches that may help to limit cost and schedule risks. (See table.)

Selected Software Development and Cybersecurity Approaches That May Limit Risks and Number of Major DOD Business IT Programs That Reported Using the Approach

Software development and cybersecurity approaches that may limit risk

Number of programs that reported using the approach

Using off-the-shelf software

19 of 22

Implementing continuous iterative software development

18 of 22

Delivering software at least every 6 monthsa

16 of 22

Developing or planning to develop a cybersecurity strategy

21 of 22

Conducting developmental cybersecurity testing

16 of 22

Conducting operational cybersecurity testing

15 of 22

Source: GAO analysis of Department of Defense questionnaire responses. | GAO-21-351
aThe Defense Innovation Board encourages more frequent delivery of working software to users for Agile and DevOps practices.

Program officials also reported facing a variety of software development challenges while implementing these approaches. These included difficulties finding and hiring staff, transitioning from waterfall to Agile software development, and managing technical environments. DOD's continued efforts to address these challenges will be critical to the department's implementation of modern software development approaches.

DOD has also made organizational and policy changes intended to improve the management of its IT acquisitions, such as taking steps to implement Agile software development and improve data transparency. In addition, to address statutory requirements, DOD has taken steps to remove the department's chief management officer (CMO) position. However, the department had not yet sufficiently implemented these changes. Officials from many of the 18 programs GAO assessed that reported using Agile development reported that DOD had implemented activities associated with Agile transition best practices to only some or little to no extent, indicating that the department had not sufficiently implemented best practices. For example, 12 of the 18 programs reported that DOD's life-cycle activities only supported Agile methods to some or little to no extent. Program officials also reported challenges associated with implementing Agile software development. The department has a variety of efforts underway to help with its implementation of Agile software development. DOD officials stated that the department's transition to Agile will take years and will require sustained engagement throughout DOD.

In addition, DOD has taken steps aimed at improving the sharing and transparency of data it uses to monitor its acquisitions. According to a November 2020 proposal from the Office of the Under Secretary for Acquisition and Sustainment, DOD officials are to develop data strategies and metrics to assess performance for the department's acquisition pathways. However, as of February 2021, DOD did not have data strategies and had not finalized metrics for the two pathways associated with the programs discussed in this report. Officials said they were working with DOD programs and components to finalize initial pathway metrics. They stated that they plan to implement them in fiscal year 2021 and continue to refine and adjust them over the coming years. Without important data from acquistion pathways and systems, DOD risks not having timely quantitative insight into program performance, including its acquisition reform efforts.

Finally, DOD's CMO position was eliminated by a statute enacted in January 2021. This position was responsible for key efforts associated with the department's business systems modernization, which has been on GAO's High Risk List since 1995. DOD plans to take steps to address the uncertainty associated with the recent elimination of the position.

Why GAO Did This Study

For fiscal year 2021, DOD requested approximately $37.7 billion for IT investments. These investments included major business IT programs, which are intended to help the department carry out key business functions, such as financial management and health care.

The National Defense Authorization Act for Fiscal Year 2019 included a provision for GAO to assess selected IT programs annually through March 2023. GAO's objectives for this review were to (1) summarize DOD's reported performance of its portfolio of IT acquisition programs and the reasons for this performance; (2) evaluate DOD's assessments of program risks; (3) summarize DOD's approaches to software development and cybersecurity and identify associated challenges; and (4) evaluate how selected organizational and policy changes could affect IT acquisitions.

To address these objectives, GAO selected 29 major business IT programs that DOD reported to the federal IT Dashboard (a public website that includes information on the performance of major IT investments) as of September 2020. GAO reviewed planned expenditures for these programs, from fiscal years 2019 through 2022, as reported in the department's FY 2021 budget request. It also aggregated program office responses to a GAO questionnaire that requested information about cost and schedule changes that occurred since January 2019 and the early impacts of COVID-19.

GAO also analyzed the risks of the 22 programs that were actively using central repositories known as risk registers to manage program risks. GAO used these registers to create program risk ratings, and then compared its ratings to those of the DOD chief information officer (CIO).

In addition, GAO aggregated DOD program office responses to the questionnaire that requested information about the software and cybersecurity practices used by 22 of the 29 IT programs that were actively developing software. GAO compared the responses to relevant guidance and leading practices.

GAO reviewed selected IT-related organizational and policy changes and reviewed reports and documentation related to the effects of these changes on IT acquisitions. GAO also aggregated program office responses to the questionnaire that requested information about DOD's implementation of these changes. This included information on DOD's implementation of best practices as part of its efforts to implement Agile software development. GAO met with relevant DOD officials to discuss each of the topics addressed in this report.


Skip to Recommendations

Recommendations

GAO is making two recommendations to DOD related to revisiting the department's CIO risk ratings and improving data strategies and automated data collection efforts for the business system and software acquisition pathways necessary for stakeholders to monitor acquisitions and critical to the department's ability to assess acquisition performance.

DOD concurred with GAO's recommendations and described actions it planned to take, or had begun taking, to address them.

Recommendations for Executive Action

Agency Affected Recommendation Status
Department of Defense The Secretary of Defense should direct the Chief Information Officer to revisit program risk ratings for its next submission to the federal IT Dashboard for the programs where the DOD CIO's program risk ratings indicated less risk than GAO's assessments of program risk. (Recommendation 1)
Open
In October 2021, the Department of Defense (DOD) described steps it has taken and plans to take to address this recommendation. Specifically, the department stated that it tasked components to review risk ratings that indicated less risk than GAO's risk ratings and to report findings to inform their component Chief Information Officers (CIOs) next risk ratings submission to DOD CIO. In addition, the department stated that it will assess component CIO risk ratings before posting final risk ratings to the Federal IT Dashboard by November 2021. GAO also has ongoing work assessing, among other things, program risk levels and we will update the status of this recommendation when appropriate. As of August 2022, DOD had not provided any updated information on the status of this recommendation.
Department of Defense The Secretary of Defense should direct the Under Secretary of Defense for Acquisition and Sustainment, in consultation with appropriate internal and external stakeholders, to ensure the data strategies and data collection efforts for the business system and software acquisition pathways define, collect, automate, and share, with the appropriate level of visibility, the metrics necessary for stakeholders to monitor acquisitions and that are critical to the department's ability to assess acquisition performance. (Recommendation 2)
Open
In October 2021, DOD described steps it has taken and plans to take to address this recommendation. Specifically, related to the business systems pathway, the department stated that it plans to determine, review, and analyze existing defense business systems data reported to the Office of the Secretary of Defense and military services; define reporting thresholds and identify metrics; and document required defense business system data elements by the fourth quarter of fiscal year 2022. Related to the software pathway, the department stated that it has established a software acquisition pathway (SWP) data collection strategy and socialized it with component headquarters and relevant program offices. The department also stated that it plans to prepare a SWP semi-annual reporting template and conduct trial submissions with early adopter programs to gain insights, implement suggestions, and improve the template. Further, the department plans to collect the first iteration of the SWP program metrics data via a MS Excel template in October 2021 and then transition to automated transmission of metrics during calendar year 2022. GAO has ongoing work assessing, among other things, actions DOD has taken to implement its plans related to this recommendation and we will continue to follow-up with the department and update the status of this recommendation as appropriate. As of August 2022, DOD had not provided any updated information on the status of this recommendation.

Full Report

GAO Contacts