Defense Acquisitions:

Stronger Management Practices Are Needed to Improve DOD's Software-Intensive Weapon Acquisitions

GAO-04-393: Published: Mar 1, 2004. Publicly Released: Mar 1, 2004.

Additional Materials:

Contact:

Katherine V. Schinasi
(202) 512-4841
contact@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

The Department of Defense (DOD) has been relying increasingly on computer software to introduce or enhance performance capabilities of major weapon systems. To ensure successful outcomes, software acquisition requires disciplined processes and practices. Without such discipline, weapon programs encounter difficulty in meeting cost and schedule targets. For example, in fiscal year 2003, DOD might have spent as much as $8 billion to rework software because of quality-related issues. GAO was asked to identify the practices used by leading companies to acquire software and to analyze the causes of poor outcomes of selected DOD programs. GAO also was asked to evaluate DOD's efforts to develop programs for improving software acquisition processes and to assess how those efforts compare with leading companies' practices.

Software developers and acquirers at firms that GAO visited use three fundamental management strategies to ensure the delivery of high-quality products on time and within budget: working in an evolutionary environment, following disciplined development processes, and collecting and analyzing meaningful metrics to measure progress. When these strategies are used together, leading firms are better equipped to improve their software development processes on a continuous basis. An evolutionary approach sets up a more manageable environment--one in which expectations are realistic and developers are permitted to make incremental improvements. The customer benefits because the initial product is available sooner and at a lower, more predictable cost. This avoids the pressure to incorporate all the desired capabilities into a single product right away. Within an evolutionary environment, there are four phases that are common to software development: setting requirements, establishing a stable design, writing code, and testing. At the end of each of these phases, developers must demonstrate that they have acquired the right knowledge before proceeding to the next development phase. To provide evidence that the right knowledge was captured, leading developers emphasize the use of meaningful metrics, which helps developers, managers, and acquirers to measure progress. These metrics focus on cost, schedule, the size of a project, performance requirements, testing, defects, and quality. In a review of five DOD programs, GAO found that outcomes were mixed for software-intensive acquisitions. The F/A-18 C/D, a fighter and attack aircraft, and the Tactical Tomahawk missile had fewer additional cost and schedule delays. For these programs, developers used an evolutionary approach, disciplined processes, and meaningful metrics. In contrast, the following programs, which did not follow these management strategies, experienced schedule delays and cost growth: F/A-22, an air dominance aircraft; Space- Based Infrared System, a missile-detection satellite system; and Comanche, a multimission helicopter. In response to congressional requirements, DOD, the military services, and the Missile Defense Agency have taken positive steps to improve the environment for acquiring software-intensive systems. However, their plans to implement software process improvement programs are not yet complete and more work is required to ensure controls that would help managers increase the chances of successful acquisition outcomes. Such controls include documenting baseline requirements agreements between the developer and acquirer that leverage systems engineering knowledge, meeting with the developer for periodic reviews (gates) during the development process, and obtaining meaningful metrics from the developer to manage the program. Furthermore, there are no assurances that program managers will be held accountable for using the plans once they are completed.

Status Legend:

More Info
  • Review Pending-GAO has not yet assessed implementation status.
  • Open-Actions to satisfy the intent of the recommendation have not been taken or are being planned, or actions that partially satisfy the intent of the recommendation have been taken.
  • Closed-implemented-Actions that satisfy the intent of the recommendation have been taken.
  • Closed-not implemented-While the intent of the recommendation has not been satisfied, time or circumstances have rendered the recommendation invalid.
    • Review Pending
    • Open
    • Closed - implemented
    • Closed - not implemented

    Recommendations for Executive Action

    Recommendation: As DOD changes the way it manages software intensive systems, it must take steps to ensure better acquisition outcomes. The Secretary of Defense should, to ensure DOD has the knowledge it needs to oversee software-intensive acquisitions, have acquirers require software contractors to collect and report metrics related to cost, schedule, size, requirements, tests, defects, and quality to program offices on a monthly basis and before program milestones and that acquirers should ensure that contractors have an earned value management system that reports cost and schedule information at a level of work that provides information specific to software development.

    Agency Affected: Department of Defense

    Status: Closed - Implemented

    Comments: DOD partially concurred with the recommendation, agreeing that the developer should provide metrics related to software development, but did not agree with the monthly intervals for reporting. The Air Force initiated a policy change requiring earned value reports on software development and a core set of metrics including those enumerated in GAO's report.

    Recommendation: As DOD changes the way it manages software intensive systems, it must take steps to ensure better acquisition outcomes. The Secretary of Defense should, to ensure DOD acquisitions are managed to a disciplined process, have acquirers develop a list of systems engineering deliverables (including software), tailored to the program characteristics, and based on the results of systems engineering activities that software developers are required to provide at the appropriate stages of the system development phases of requirements, design, fabrication/coding, integration, and testing.

    Agency Affected: Department of Defense

    Status: Closed - Implemented

    Comments: DOD concurred with the recommendation and the Air Force initiated a policy change in September 2004 to require software deliverables and to ensure developer maturity.

    Recommendation: As DOD changes the way it manages software intensive systems, it must take steps to ensure better acquisition outcomes. The Secretary of Defense should, to assure DOD appropriately sets and manages requirements, document that software requirements are achievable based on knowledge obtained from systems engineering prior to beginning development and that DOD and the contractor have a mutual understanding of the software requirements. Furthermore, trade-off analyses should be performed, supported by systems engineering analysis, considering performance, cost, and schedule impacts of major changes to software requirements.

    Agency Affected: Department of Defense

    Status: Closed - Implemented

    Comments: DOD concurred with the recommendation, and in September 2004 the Air Force issued a policy memorandum to its acquisition staff to produce realistic baseline performance requirements based on systems engineering analysis and establish effective program office processes to support software development including ensuring that the developer has the skills and strengths to manage development.

    Recommendation: As DOD changes the way it manages software intensive systems, it must take steps to ensure better acquisition outcomes. These practices should be included and enforced with controls and incentives in DOD's acquisitions policy, software acquisition improvement plans and development contracts, by the Secretary of Defense.

    Agency Affected: Department of Defense

    Status: Closed - Implemented

    Comments: The Air Force created a policy memorandum in September 2004 to incorporate improvements to software acquisition and development into its policy.

    Apr 18, 2014

    Apr 16, 2014

    Apr 11, 2014

    Apr 10, 2014

    Apr 9, 2014

    Apr 8, 2014

    Apr 3, 2014

    Looking for more? Browse all our products here