DOD Space Acquisitions: Including Users Early and Often in Software Development Could Benefit Programs
Fast Facts
Developing software for DOD space systems, like GPS, has historically taken longer and cost billions of dollars more than planned.
We looked at four software-intensive DOD space systems that had cost growth or delays.
While DOD has started using better software development approaches, we found some challenges to making them work. For example, program offices and system developers don't consistently involve the systems' end users in development, making it hard to ensure that the systems will meet their needs.
We recommended DOD provide specific direction on when and how often to involve users, and document the involvement to ensure it happens.
This space system relies on software to track objects in Earth's orbit
Military personnel looking at a bank of large monitors in a dimly lit room.
Highlights
What GAO Found
The four major Department of Defense (DOD) software-intensive space programs that GAO reviewed struggled to effectively engage system users. These programs are the Air Force's Joint Space Operations Center Mission System Increment 2 (JMS), Next Generation Operational Control System (OCX), Space-Based Infrared System (SBIRS); and the Navy's Mobile User Objective System (MUOS). These ongoing programs are estimated to cost billions of dollars, have experienced overruns of up to three times originally estimated cost, and have been in development for periods ranging from 5 to over 20 years. Previous GAO reports, as well as DOD and industry studies, have found that user involvement is critical to the success of any software development effort. For example, GAO previously reported that obtaining frequent feedback is linked to reducing risk, improving customer commitment, and improving technical staff motivation. However, the programs GAO reviewed often did not demonstrate characteristics of effective user engagement that are identified in DOD policy and statute:
- Early engagement. OCX involved users early; JMS planned to but, in practice, did not; SBIRS and MUOS did not plan to involve users early.
- Continual engagement. JMS, OCX, and SBIRS all planned to continually involve users but, in practice, did not fully do so; MUOS did not plan to do so.
- Feedback based on actual working software. OCX and SBIRS provided users opportunities to give such feedback but only years into software development; JMS and MUOS did not provide opportunities for feedback.
- Feedback incorporated into subsequent development. JMS, OCX, and SBIRS all planned to incorporate user feedback but, in practice, have not done so throughout development; MUOS did not plan to do so.
As reflected above, actual program efforts to involve users and obtain and incorporate feedback were often unsuccessful. This was due, in part, to the lack of specific guidance on user involvement and feedback. Although DOD policies state that users should be involved and provide feedback on software development projects, they do not provide specific guidance on the timing, frequency, and documentation of such efforts. Without obtaining user feedback and acceptance, programs risk delivering systems that do not meet users' needs. In selected instances, the lack of user involvement has contributed to systems that were later found to be operationally unsuitable.
The programs GAO reviewed also faced software-specific challenges in using commercial software, applying outdated software tools, and having limited knowledge and training in newer software development techniques. For example, programs using commercial software often underestimated the effort required to integrate such software into an overall system. Secondly, selected programs relied on obsolete software tools that they were accustomed to using but which industry had since replaced. Finally, GAO found that two of the reviewed programs lacked knowledge of more modern software development approaches. DOD has acknowledged these challenges and has efforts underway to address each of them.
Why GAO Did This Study
Over the next 5 years, DOD plans to spend over $65 billion on its space system acquisitions portfolio, including many systems that rely on software for key capabilities. However, software-intensive space systems have had a history of significant schedule delays and billions of dollars in cost growth.
Senate and House reports accompanying the National Defense Authorization Act for Fiscal Year 2017 contained provisions for GAO to review challenges in software-intensive DOD space programs. This report addresses, among other things, (1) the extent to which these programs have involved users; and (2) what software-specific management challenges, if any, programs faced.
To do this work, GAO reviewed four major space defense programs with cost growth or schedule delays caused, in part, by software. GAO reviewed applicable statutes and DOD policies and guidance that identified four characteristics of effective user engagement. GAO reviewed program documentation; and interviewed program officials, contractors, and space systems users. GAO also analyzed program metrics, test and evaluation reports, and external program assessments.
Recommendations
GAO is making two recommendations that DOD ensure its guidance that addresses software development provides specific, required direction on the timing, frequency, and documentation of user involvement and feedback. DOD concurred with the recommendations.
Recommendations for Executive Action
Agency Affected | Recommendation | Status |
---|---|---|
Department of Defense | The Secretary of Defense should ensure the department's guidance that addresses software development provides specific, required direction on when and how often to involve users so that such involvement is early and continues through the development of the software and related program components. (Recommendation 1) |
DOD concurred with this recommendation. In October 2020, DOD issued its software acquisition pathway guidance, DOD Instruction 5000.87. The guidance requires programs to actively engage users throughout the software lifecycle, including the planning phase; generate a user agreement that ensures continuous user involvement; and commits resourcing of operational users to provide feedback.
|
Department of Defense | The Secretary of Defense should ensure the department's guidance that addresses software development provides specific, required direction on documenting and communicating user feedback to stakeholders during software system development. (Recommendation 2) |
DOD concurred with this recommendation. In October 2020, DOD issued its software acquisition pathway guidance, DOD Instruction 5000.87. The guidance requires programs to solicit user feedback throughout the software lifecycle, incorporate the feedback into the product roadmap and priorities backlog, and update program strategies and designs based on user feedback.
|