Developing software for DOD space systems, like GPS, has historically taken longer and cost billions of dollars more than planned.

We looked at four software-intensive DOD space systems that had cost growth or delays.

While DOD has started using better software development approaches, we found some challenges to making them work. For example, program offices and system developers don't consistently involve the systems' end users in development, making it hard to ensure that the systems will meet their needs.

We recommended DOD provide specific direction on when and how often to involve users, and document the involvement to ensure it happens.

This space system relies on software to track objects in Earth's orbit

Military personnel looking at a bank of large monitors in a dimly lit room.

• View Highlights

What GAO Found

The four major Department of Defense (DOD) software-intensive space programs that GAO reviewed struggled to effectively engage system users. These programs are the Air Force's Joint Space Operations Center Mission System Increment 2 (JMS), Next Generation Operational Control System (OCX), Space-Based Infrared System (SBIRS); and the Navy's Mobile User Objective System (MUOS). These ongoing programs are estimated to cost billions of dollars, have experienced overruns of up to three times originally estimated cost, and have been in development for periods ranging from 5 to over 20 years. Previous GAO reports, as well as DOD and industry studies, have found that user involvement is critical to the success of any software development effort. For example, GAO previously reported that obtaining frequent feedback is linked to reducing risk, improving customer commitment, and improving technical staff motivation. However, the programs GAO reviewed often did not demonstrate characteristics of effective user engagement that are identified in DOD policy and statute:

• Early engagement. OCX involved users early; JMS planned to but, in practice, did not; SBIRS and MUOS did not plan to involve users early.

• Continual engagement. JMS, OCX, and SBIRS all planned to continually involve users but, in practice, did not fully do so; MUOS did not plan to do so.

• Feedback based on actual working software. OCX and SBIRS provided users opportunities to give such feedback but only years into software development; JMS and MUOS did not provide opportunities for feedback.

• Feedback incorporated into subsequent development. JMS, OCX, and SBIRS all planned to incorporate user feedback but, in practice, have not done so throughout development; MUOS did not plan to do so.

As reflected above, actual program efforts to involve users and obtain and incorporate feedback were often unsuccessful. This was due, in part, to the lack of specific guidance on user involvement and feedback. Although DOD policies state that users should be involved and provide feedback on software development projects, they do not provide specific guidance on the timing, frequency, and documentation of such efforts. Without obtaining user feedback and acceptance, programs risk delivering systems that do not meet users' needs. In selected instances, the lack of user involvement has contributed to systems that were later found to be operationally unsuitable.

The programs GAO reviewed also faced software-specific challenges in using commercial software, applying outdated software tools, and having limited knowledge and training in newer software development techniques. For example, programs using commercial software often underestimated the effort required to integrate such software into an overall system. Secondly, selected programs relied on obsolete software tools that they were accustomed to using but which industry had since replaced. Finally, GAO found that two of the reviewed programs lacked knowledge of more modern software development approaches. DOD has acknowledged these challenges and has efforts underway to address each of them.

Why GAO Did This Study

Over the next 5 years, DOD plans to spend over $65 billion on its space system acquisitions portfolio, including many systems that rely on software for key capabilities. However, software-intensive space systems have had a history of significant schedule delays and billions of dollars in cost growth.

Senate and House reports accompanying the National Defense Authorization Act for Fiscal Year 2017 contained provisions for GAO to review challenges in software-intensive DOD space programs. This report addresses, among other things, (1) the extent to which these programs have involved users; and (2) what software-specific management challenges, if any, programs faced.

To do this work, GAO reviewed four major space defense programs with cost growth or schedule delays caused, in part, by software. GAO reviewed applicable statutes and DOD policies and guidance that identified four characteristics of effective user engagement. GAO reviewed program documentation; and interviewed program officials, contractors, and space systems users. GAO also analyzed program metrics, test and evaluation reports, and external program assessments.

What GAO Recommends

GAO is making two recommendations that DOD ensure its guidance that addresses software development provides specific, required direction on the timing, frequency, and documentation of user involvement and feedback. DOD concurred with the recommendations.

For more information, contact Jon Ludwigson at (202) 512-4841 or ludwigsonj@gao.gov.