Skip to main content

Defense Acquisitions: Major Weapon Systems Continue to Experience Cost and Schedule Problems under DOD's Revised Policy

GAO-06-368 Published: Apr 14, 2006. Publicly Released: Apr 14, 2006.
Jump To:
Skip to Highlights

Highlights

The Department of Defense (DOD) is planning to invest $1.3 trillion between 2005 and 2009 in researching, developing, and procuring major weapon systems. How DOD manages this investment has been a matter of congressional concern for years. Numerous programs have been marked by cost overruns, schedule delays, and reduced performance. Over the past 3 decades, DOD's acquisition environment has undergone many changes aimed at curbing cost, schedule, and other problems. In order to determine if the policy DOD put in place is achieving its intended goals, we assessed the outcomes of major weapons development programs initiated under the revised policy. Additionally, we assessed whether the policy's knowledge-based, evolutionary principles are being effectively implemented, and whether effective controls and specific criteria are in place and being used to make sound investment decisions.

Recommendations

Recommendations for Executive Action

Agency Affected Recommendation Status
Department of Defense DOD must ensure that appropriate knowledge is captured and used at critical junctures to make decisions about moving a program forward and investing more money. The Secretary of Defense should require program officials to demonstrate that they have captured appropriate knowledge at three key points--program start, design review for transitioning from system integration to system demonstration, and production commitment--as a condition for investing resources. At a minimum those controls should require program officials to demonstrate that they have achieved a level of knowledge that meets or exceeds the following criteria for Program start (Milestone B): Start of product development: (1) demonstrate technologies to high readiness levels; (2) ensure that requirements for the product are informed by the systems-engineering process; (3) establish cost and schedule estimates for product on the basis of knowledge from preliminary design using system-engineering tools; and (4) conduct decision review for program start.
Closed – Implemented
Title 10 U.S.C. section 2366a, enacted by sec. 801 of the National Defense Authorization Act for fiscal year 2006 (P.L. 109-163, January 2006) and Title 10 section 2366b, enacted by sec. 943 of the National Defense Authorization Act for fiscal year 2008, establish controls to ensure appropriate knowledge is captured prior to beginning system development. Both sections of the law require the Secretary of Defense to certify that major defense acquisition programs have certain knowledge based deliverables and meet certain knowledge based criteria prior to the start of system development at milestone B (section 2366a establishes controls for milestone B, while section 2366b establishes controls for milestone A). In addition, the USD (AT&L) issued a policy memorandum in September 2007 that requires all pending and future acquisition programs to have acquisition strategies and funding to provide for two or more competing teams to produce prototypes through milestone B. The stated intent of this prototyping policy is to reduce technical risk, validate designs, validate cost estimates, evaluate manufacturing processes, and refine requirements. We believe that the certification requirements in the law and the new prototyping policy establish controls that meet the intent of our recommendation.
Department of Defense DOD must ensure that appropriate knowledge is captured and used at critical junctures to make decisions about moving a program forward and investing more money. The Secretary of Defense should require program officials to demonstrate that they have captured appropriate knowledge at three key points--program start, design review for transitioning from system integration to system demonstration, and production commitment--as a condition for investing resources. At a minimum those controls should require program officials to demonstrate that they have achieved a level of knowledge that meets or exceeds the following criteria for Design readiness review: Beginning of system demonstration: (1) complete 90 percent of design drawings; (2) complete subsystem and system design reviews; (3) demonstrate with prototype that design meets requirements; (4) obtain stakeholders' concurrence that drawings are complete and producible; (5) complete the failure modes and effects analysis; (6) identify key system characteristics; (7) identify critical manufacturing processes; (8) establish reliability targets and growth plan on the basis of demonstrated reliability rates of components and subsystems; and (9) conduct decision review to enter system demonstration.
Closed – Implemented
DOD has initiated a number of acquisition reform pilot programs and issued several policy memorandums, which ultimately resulted in the revision of DOD's acquisition policy, known as the DOD 5000 series. Although more could be done to further strengthen the policy, we believe that the most recent revision, issued in December 2008, essentially addresses the intent of our recommendation. The revised policy requires program managers to hold a Preliminary Design Review (PDR) prior to initiating system development--now referred to as Engineering and Manufacturing Development (EMD). According to the policy a successful PDR will inform requirements trades; improve cost estimation; and identify remaining design, integration, and manufacturing risks. In addition, the policy also emphasizes the importance of using key metrics to assess design maturity at the system-level Critical Design Review (CDR) before continuing into the demonstration phase of EMD--now referred to as System Capability and Manufacturing Process Demonstration. The metrics emphasized are: successful completion of subsystem CDRs; the percentage of hardware and software product build-to specifications and drawings completed and under configuration management; planned corrective actions to hardware/software deficiencies; adequate developmental testing; an assessment of environment, safety and occupational health risks; a completed failure modes and effects analysis; the identification of key system characteristics; the maturity of critical manufacturing processes; and an estimate of system reliability based on demonstrated reliability rates. In a briefing given after the revised policy was issued, DOD officials detailed more specific metrics that they noted are required by the DOD Acquisition process, and will be applied to new and on-going programs. The metrics related to CDR are (1) completion of System-level CDR, (2) completion of 90 percent of design drawings, (3) successful milestone decision review conducted, (4) completion of failure modes and effects analysis, and (5) establishment of reliability targets and identification of key manufacturing processes.
Department of Defense DOD must ensure that appropriate knowledge is captured and used at critical junctures to make decisions about moving a program forward and investing more money. The Secretary of Defense should require program officials to demonstrate that they have captured appropriate knowledge at three key points--program start, design review for transitioning from system integration to system demonstration, and production commitment--as a condition for investing resources. At a minimum those controls should require program officials to demonstrate that they have achieved a level of knowledge that meets or exceeds the following criteria for Production commitment (Milestone C): Initiation of low-rate production: (1) demonstrate manufacturing processes; (2) build production-representative prototypes; (3) test production-representative prototypes to achieve reliability goal; (4) test production-representative prototypes to demonstrate product in operational environment; (5) collect statistical process control data; (6) demonstrate that critical processes are capable and in statistical control; and (7) conduct decision review to begin production.
Closed – Implemented
DOD has revised its defense acquisition policy and issued a Manufacturing Readiness Level (MRL) Deskbook. While DOD's revised policy, issued in December 2008, does not strictly require programs to collect or use statistical process control metrics, it has placed increased emphasis on the importance of building and testing production representative prototypes, and on the importance of assessing and measuring manufacturing readiness before beginning production. DOD's policy states that entrance into the production and deployment phase at Milestone C depends on the following criteria: acceptable performance in developmental test and evaluation and operational assessment; mature software capability; no significant manufacturing risks; manufacturing processes under control; an approved Initial Capabilities Document; an approved Capability Production Document; a refined integrated architecture; acceptable interoperability; acceptable operational supportability; and demonstration that the system is affordable throughout the life cycle, fully funded, and properly phased for rapid acquisition. In a briefing given after the revised policy was issued, DOD officials detailed more specific metrics that they noted are required by the DOD Acquisition process and will be applied to new and on-going programs. The metrics related to Milestone C are: (1) Joint Requirements Oversight Council-validation of the program requirements documentation related to production; (2) demonstration of manufacturing processes; and (3) demonstration that production-representative test articles have met the approved requirements in the system's intended environment. Subsequently, in May 2011, the Department issued its MRL Deskbook, which is intended to provide best practices for conducting assessments of manufacturing readiness. DOD expects the Deskbook to be used as a tool and resource to help defense programs with manufacturing related reporting and oversight. The collective impact of these efforts to place added emphasis on the need for manufacturing maturity before committing to a production program is sufficient to meet the intent of our recommendation.
Department of Defense Furthermore, to ensure that major decisions are transparent and that program officials and decision makers are held accountable, the Secretary of Defense should require decision makers to include written rationale for each major decision in acquisition decision documentation. The rationale should address the key knowledge-based criteria appropriate for milestone decisions, explain why a program's level of knowledge in each area was deemed acceptable if criteria have not been met and provide a plan for achieving the knowledge necessary to meet criteria within a given time frame.
Closed – Implemented
The Department of Defense agreed that they should be accountable for program decisions and appropriately record the considerations in moving from one stage of development to the next. Milestone decision authorities are to employ the decision memorandums to document decisions and their rationale. Title 10 U.S.C. section 2366a, enacted by sec. 801 of the National Defense Authorization Act for fiscal year 2006 (P.L. 109-163, January 2006) and Title 10 section 2366b, enacted by sec. 943 of the National Defense Authorization Act for fiscal year 2008, establish controls to ensure appropriate knowledge is captured prior to beginning system development. Both sections of the law require the Secretary of Defense to certify that major defense acquisition programs have certain knowledge based deliverables and meet certain knowledge based criteria prior to the start of system development at milestone B (section 2366a establishes controls for milestone B, while section 2366b establishes controls for milestone A). We believe that these certification requirements establish controls that meet the intent of our recommendation. Therefore, we are closing this recommendation as implemented.

Full Report

GAO Contacts

Office of Public Affairs

Topics

Cost overrunsDefense cost controlDefense procurementInternal controlsMilitary policiesPerformance measuresPolicy evaluationProcurement planningProcurement policySchedule slippagesWeapons research and developmentWeapons systemsInvestment management