This is the accessible text file for GAO report number GAO-06-368 
entitled 'Defense Acquisitions: Major Weapon Systems Continue to 
Experience Cost and Schedule Problems under DOD's Revised Policy' which 
was released on April 14, 2006.

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as part 
of a longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov.

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately.

Report to Congressional Committees:

United States Government Accountability Office:

GAO:

April 2006:

Defense Acquisitions:

Major Weapon Systems Continue to Experience Cost and Schedule Problems 
under DOD's Revised Policy:

GAO-06-368:

GAO Highlights:

Highlights of GAO-06-368, a report to congressional committees.

Why GAO Did This Study:

The Department of Defense (DOD) is planning to invest $1.3 trillion 
between 2005 and 2009 in researching, developing, and procuring major 
weapon systems. How DOD manages this investment has been a matter of 
congressional concern for years. Numerous programs have been marked by 
cost overruns, schedule delays, and reduced performance. Over the past 
3 decades, DODís acquisition environment has undergone many changes 
aimed at curbing cost, schedule, and other problems. In order to 
determine if the policy DOD put in place is achieving its intended 
goals, we assessed the outcomes of major weapons development programs 
initiated under the revised policy. Additionally, we assessed whether 
the policyís knowledge-based, evolutionary principles are being 
effectively implemented, and whether effective controls and specific 
criteria are in place and being used to make sound investment decisions.

What GAO Found:

Changes made in DODís acquisition policy over the past 5 years have not 
eliminated cost and schedule problems for major weapons development 
programs. Of the 23 major programs we assessed, 10 are already 
expecting development cost overruns greater than 30 percent or have 
delayed the delivery of initial operational capability to the 
warfighter by at least 1 year. The overall impact of these costly 
conditions is a reduction in the value of DODís defense dollars and a 
lower return on investment. The following table illustrates the problem.

Table: Cost and Schedule Outcomes Sorted by Percent of Product 
Development Remaining.

Programs: Aerial Common Sensor;
Percent Cost Growth[A]: 45%
Schedule growth in months: 24;
Percent of development remaining: 85%.

Programs: Future Combat System;
Percent Cost Growth[A]: 48%
Schedule growth in months: 48;
Percent of development remaining: 78%.

Programs: Joint Strike Fighter;
Percent Cost Growth[A]: 30%
Schedule growth in months: 23;
Percent of development remaining: 60%.

Programs: Expeditionary Fighting Vehicle;
Percent Cost Growth[A]: 61%
Schedule growth in months: 48;
Percent of development remaining: 49%.

Programs: C-130 Avionics Fighting Vehicle;
Percent Cost Growth[A]: 122%
Schedule growth in months: Delays anticipated;
Percent of development remaining: Undetermined.

Programs: Global Hawk (RQ-4B);
Percent Cost Growth[A]: 166%
Schedule growth in months: Delays Anticipated;
Percent of development remaining: Undetermined.

[A] Cost growth is expressed as the percent change in program 
development cost estimates in 2005 base year dollars.

Source: DOD (data); GAO (analysis and presentation)

[End to table]

Poor execution of the revised acquisition policy is a major cause of 
DODís continued problems. DOD frequently bypasses key steps of the 
knowledge-based process outlined in the policy, falls short of 
attaining key knowledge, and continues to pursue revolutionaryórather 
than evolutionary or incrementalóadvances in capability. Nearly 80 
percent of the programs GAO reviewed did not fully follow the knowledge-
based process to develop a sound business case before committing to 
system development. Most of the programs we reviewed started system 
development with immature technologies, and half of the programs that 
have held design reviews did so before achieving a high level of design 
maturity. These practices increase the likelihood that problems will be 
discovered late in development when they are more costly to address. 
Furthermore, DODís continued pursuit of revolutionary leaps in 
capability also runs counter to the policyís guidance.

DOD has not closed all of the gaps in the policy that GAO identified 
nearly 3 years ago, particularly with regard to adding controls and 
criteria. Effective controls require decision makers to measure 
progress against specific criteria and ensure that managers capture key 
knowledge before moving to the next acquisition phase. However, DODís 
policy continues to allow managers to approach major investment 
decisions with many unknowns. Without effective controls that require 
program officials to satisfy specific criteria, it is difficult to hold 
decision makers or program managers accountable to cost and schedule 
targets. In this environment, decision-making transparency is crucial, 
but DOD is lacking in this area as well.

What GAO Recommends:

GAO recommends that DOD insert specific criteria into the policy at key 
investment points and require programs satisfy those criteria before 
allowing them to move forward. In order to insure transparency and 
accountability, GAO also recommends that DOD require decision makers to 
include the rationale for their decisions in decision documentation. 
DOD partially concurred with our recommendations.

To view the full product, including the scope and methodology, click on 
the link above. For more information, contact Michael J. Sullivan at 
(202) 512-4841 or sullivanm@gao.gov.

[End of Section]

Contents:

Letter:

Results in Brief:

Background:

DOD's Revised Policy Has Not Improved Development Program Outcomes:

DOD Is Not Effectively Implementing the Policy's Knowledge-Based, 
Evolutionary Approach:

Specific Criteria Are Needed to Ensure Disciplined and Transparent 
Investment Decisions:

Conclusions:

Recommendations for Executive Action:

Agency Comments and Our Evaluation:

Appendix I: Scope and Methodology:

Appendix II: Comments from the Department of Defense:

Appendix III: Program Data for 23 Programs Initiated under DOD's 
Revised Acquisition Policy (as of December 2005):

Related GAO Products:

Tables:

Table 1: Cost and Schedule Outcomes for 23 Programs Initiated under the 
Revised Policy (as of December 2005):

Table 2: Cost and Schedule Outcomes for 6 of the 10 Largest Development 
Programs Sorted by Percent of System Development Remaining:

Table 3: Assessment of Program Design Maturity:

Table 4: Assessment of Program Acquisition Strategies for GAO's Nine 
Case Studies:

Table 5: Types of Controls Considered Best Practices for Successful 
Product Development:

Figures:

Figure 1: Development Cost Overruns by Decade (in Fiscal Year 2005 
Dollars) and Key Reform Efforts:

Figure 2: Comparison of DOD's Revised Policy and Commercial Best 
Practices Model:

Figure 3: Cost and Schedule Growth under DOD's Revised Policy:

Figure 4: Comparison of Programs with Mature versus Immature 
Technologies at Start of System Development:

Abbreviations:

DOD: Department of Defense:

GAO: Government Accountability Office:

United States Government Accountability Office:

Washington, DC 20548:

April 13, 2006:

The Honorable John Ensign:
Chairman:
The Honorable Daniel K. Akaka:
Ranking Minority Member:
Subcommittee on Readiness and Management Support Committee on Armed 
Services:
United States Senate:

The Honorable Duncan L. Hunter:
Chairman:
Committee on Armed Services:
House of Representatives:

DOD's planned investment in research, development, and procurement of 
major weapon systems will total approximately $1.3 trillion between 
2005 and 2009, with over $800 billion of that investment yet to be 
made. DOD is facing a significant number of problems in managing its 
acquisitions. Military operations in Afghanistan and Iraq are consuming 
a large share of DOD resources and causing the department to invest 
more money sooner than expected to replace or fix existing weapons. 
Meanwhile, DOD is intent on transforming military operations while 
pursuing multiple megasystems that are expected to be the most 
expensive and complex ever. These costly conditions coupled with 
increases in spending for other national priorities, such as health 
care and social security, make it essential that DOD effectively 
leverage its investments, particularly in weapon system acquisitions. 
If DOD manages its current portfolio of weapons within traditional 
margins of error, the financial consequences could be dire.

DOD's strategy for acquiring major weapon systems has traditionally 
been to plan programs that would achieve a big leap forward in 
capability within a single development program, a strategy that often 
results in major cost and schedule problems. We have assessed weapon 
acquisitions as a high-risk area for 15 years, and although U.S. 
weapons are among the best in the world, the programs to acquire them 
have continued to produce poor cost and schedule outcomes. However, the 
current defense acquisition environment continues to be characterized 
by cost and schedule growth, a lack of confidence by congressional and 
DOD leaders, and no appreciable improvement in the defense acquisition 
system. DOD knows what to do to achieve better outcomes. It has written 
into policy an approach that advocates that adequate knowledge be 
attained at critical junctures before DOD managers agree to invest more 
money in the next phase of weapon system development. The policy also 
emphasizes evolutionary principles for acquiring weapons rather than 
trying to achieve a big leap forward in capability within a single 
development program. We have reported in the past that DOD's revised 
policy does not incorporate adequate controls to ensure the effective 
implementation of a knowledge-based, evolutionary acquisition process. 
However, DOD believes that the policy includes the necessary controls 
to achieve effective outcomes.

You requested that we evaluate DOD's compliance with and implementation 
of its revised acquisition policy intended to produce better cost, 
schedule, and performance outcomes for major acquisition programs. In 
order to obtain an early assessment of the cost and schedule impact of 
the revised policy, and to assess DOD's effectiveness in implementing a 
knowledge-based, evolutionary acquisition approach we assessed (1) the 
cost and schedule status of major weapons development programs 
initiated under the revised policy, (2) whether the policy's knowledge- 
based, evolutionary acquisition principles are being effectively 
implemented, and (3) whether effective controls and specific criteria 
are in place and being used to make sound investment decisions.

In conducting our evaluation, we reviewed pertinent acquisition 
statutes, policies, and guidance; analyzed development cost and 
schedule data for 23 major acquisition programs approved to start 
system development under DOD's revised acquisition policy between 
October 2000 and December 2004; conducted case study reviews of nine of 
those 23 programs; and interviewed officials from the Office of 
Secretary of Defense and each of the military services. We conducted 
our review from May 2005 to February 2006 in accordance with generally 
accepted government auditing standards. Additional information about 
our methodology is contained in appendix I.

Results in Brief:

DOD's revised acquisition policy has not led to improved acquisition 
program outcomes. Programs initiated under the revised policy are 
already beginning to experience cost and schedule problems similar to 
programs managed under prior versions of the policy. Although the 
programs we reviewed have been in development for only a short period 
of time, nearly half are already estimating development cost growth 
greater than 30 percent or are expecting to delay initial delivery to 
the warfighter by at least 1 year. Program officials are facing the 
familiar predicament of having to add unplanned money or time or to 
reduce system capabilities and quantities after initial business cases 
have been approved and system development has begun. As a result, DOD 
is reducing its buying power and is not achieving the return on 
investment it expected when the programs began.

Acquisition officials are not effectively implementing the revised 
acquisition policy's knowledge-based process. They regularly bypass key 
phases of the early acquisition process, approach key decision points 
with limited knowledge about critical technologies and system design, 
and do not employ evolutionary acquisition principles. Nearly 80 
percent of the programs we assessed were permitted to bypass the 
policy's initial major decision review and the associated systems- 
engineering process that are intended to ensure that a system's 
requirements match available resources and that a sound business case 
is developed prior to starting system development. By not consistently 
following key processes and strategies, acquisition officials are not 
ensuring that a solid foundation of knowledge about cost, schedule, and 
performance is established before allowing programs to start system 
development, thus resulting in unexecutable business cases. Although 
the policy explicitly states that programs shall increase program 
knowledge by maturing technologies before beginning system development, 
we found that almost three-fourths of the programs started since the 
policy was revised began development with immature critical 
technologies. Our analysis also indicates that decision makers are 
continuing to commit programs to system demonstration and initial 
manufacturing before officials have demonstrated high levels of design 
knowledge, as emphasized in the policy. In addition, programs like the 
Joint Strike Fighter and Future Combat System are still structured to 
achieve major leaps in capability within a single development program, 
a strategy that has historically proven to be problematic in terms of 
cost and schedule outcomes.

Effective implementation of the revised policy is limited by the 
absence of effective controls that require compliance and specific 
criteria for clearly demonstrating that acceptable levels of knowledge 
about technology, design, and manufacturing have been attained at 
critical junctures during system development before making further 
investments in a program. Without effective controls, the policy cannot 
prevent DOD decision makers from starting system development even when 
they face significant unknowns about technology, design, and 
production. Without specific criteria--or standards against which a 
judgment or decision is quantifiably based--decision makers are 
permitted to make decisions on the basis of subjective judgment. We 
reported this condition in 2003, yet DOD has not closed gaps in the 
policy. In the absence of such controls and criteria, DOD faces the 
added problems of transparency and accountability because it often does 
not sufficiently document the rationale for its decisions to allow 
acquisition programs to advance with low levels of technology, design, 
and manufacturing knowledge.

This report contains recommendations that DOD require programs to meet 
specific knowledge-based criteria at each key decision points in the 
acquisition process and require decision makers to provide clear and 
specific rationale for their decisions. In addition, we recommend that 
before programs enter system development they should be required to 
complete disciplined concept and technology development phases that 
include specific activities dedicated to capturing knowledge critical 
to developing an executable business. DOD partially concurred with our 
recommendations. DOD agrees that knowledge-based decision making is 
consistent with sound business practice and stated that it would 
continue to develop policy that reflects a knowledge-based approach and 
improves acquisition outcomes. DOD also agrees that acquisition 
decisions should be documented, decision makers should be held 
accountable, and that they should provide the rationale for their 
decisions.

Background:

Historically, DOD's programs for acquiring major weapon systems have 
taken longer, cost more, and often delivered fewer quantities and other 
capabilities than planned. GAO has documented these problems for 
decades. In 1970, GAO reported that considerable cost growth had been 
and was continuing to occur on many current development programs. Since 
that report was issued, numerous changes have been made to DOD's 
acquisition process and environment to try to improve acquisition 
outcomes. Those changes include numerous executive branch initiatives 
and legislative actions as well as roughly 11 revisions to DOD's 
acquisition policy between 1971 and 2005. Despite these efforts, 
defense acquisition programs in the past 3 decades continued to 
routinely experience cost overruns, schedule slips, and performance 
shortfalls.

Figure 1 illustrates the continued problem of development cost 
overruns. The figure depicts the combined cost overruns for large 
development programs (programs totaling more than $1 billion for 
research, development, testing and evaluation in fiscal year 2005 
dollars) in each of the past 3 decades. The figure also identifies some 
of the major studies and improvement efforts initiated during this time 
frame. As the figure illustrates, efforts to improve acquisition 
outcomes have not been successful in curbing acquisition cost problems. 
Programs initiated in the 1970s exceeded DOD's initial investment 
estimate by 30 percent, or $13 billion (in fiscal year 2005 dollars), 
and similar outcomes continued during the subsequent decades despite 
numerous reform efforts and policy revisions.

Figure 1: Development Cost Overruns by Decade (in Fiscal Year 2005 
Dollars) and Key Reform Efforts:

[See PDF for image]

Source: DOD (data); GAO (analysis and presentation)

[End of figure]

Since the mid-1990s, we have studied the best practices of leading 
commercial companies. Taking into account the differences between 
commercial product development and weapons acquisitions, we articulated 
a best practices product development model that relies on increasing 
knowledge when developing new products, separating technology 
development from product development, and following an evolutionary or 
incremental product development approach. This knowledge-based approach 
requires developers to make investment decisions on the basis of 
specific, measurable levels of knowledge at critical junctures before 
investing more money and before advancing to the next phase of 
acquisition. An evolutionary product development process defines the 
individual increments on the basis of mature technologies and a 
feasible design that are matched with firm requirements. Each increment 
should be managed as a separate and distinct acquisition effort with 
its own cost, schedule and performance baseline. An increment that 
excludes one of these key elements puts an extra burden on decision 
makers and provides a weak foundation for making development cost and 
schedule estimates. The knowledge-based, evolutionary approach in our 
model is intended to help reduce development risks and to achieve 
better program outcomes on a more consistent basis.

Hoping to improve acquisition outcomes, DOD leaders initiated 
significant revisions to the department's acquisition policy again in 
October 2000, by adopting the knowledge-based, evolutionary system 
development approach.[Footnote 1] We reported in November 2003, that 
much of the revised policy agrees with GAO's extensive body of work and 
that of successful commercial firms. DOD's revised policy emphasizes 
the importance of and provides a good framework for capturing knowledge 
about critical technologies, product design, and manufacturing 
processes. If properly implemented and enforced this approach could 
help DOD's decision makers gain the confidence they need to make 
significant and sound investment decisions for major weapon systems. 
Furthermore, the policy's emphasis on evolutionary system development 
sets up a more manageable environment for achieving knowledge. We also 
noted that DOD's policy strongly suggests the separation of technology 
development from system development, a best practice that helps reduce 
technological risk at the start of a program and makes cost and 
delivery estimates much more predictable.[Footnote 2] Figure 2 depicts 
in general how DOD's revised policy adopts key aspects of the best 
practices model.

Figure 2: Comparison of DOD's Revised Policy and Commercial Best 
Practices Model:

[See PDF for image]

Source: DOD (data); GAO (analysis and presentation)

[End of figure]

Although DOD took significant steps in the right direction, its policy 
does not include controls that require program officials to meet the 
key criteria that we believe are necessary for ensuring that acceptable 
levels of knowledge are actually captured before making additional 
significant investments. We previously recommended that DOD design and 
implement necessary controls to ensure that appropriate knowledge is 
captured and used to make decisions about moving a program forward and 
investing more money at critical junctures. DOD officials acknowledged 
the advantages of using knowledge-based controls, but stated that they 
believed the policy already included enough controls to achieve 
effective program results. The officials agreed to monitor the 
acquisition process to assess the effectiveness of those controls and 
to determine whether additional ones are necessary.

DOD's Revised Policy Has Not Improved Development Program Outcomes:

The cost and schedule outcomes being achieved by development programs 
initiated since DOD first issued its revised policy have not improved 
over those achieved by programs managed under prior versions of the 
policy. Of the 23 major programs we assessed, 10 have already reported 
estimated development cost growth greater than 30 percent or expected 
delays of at least 1 year in delivery of an initial operational 
capability to the warfighter. These programs combined represent a cost 
increase of $23 billion (in fiscal year 2005 dollars) and an average 
delay in delivery of initial capability of around 2 years. Most of the 
other programs were still in the early stages as of December 2005 with 
over half of system development remaining and had not yet reported an 
adequate amount of cost or schedule data to effectively analyze their 
progress. Table 1 contains the cost and schedule increases for the 23 
programs we assessed, expressed as a percentage of each program's 
development estimate.

Table 1: Cost and Schedule Outcomes for 23 Programs Initiated under the 
Revised Policy (as of December 2005):

Program: Expeditionary Fighting Vehicle;
Percent growth in estimated development cost[A]: 61%;
Percent growth in estimated development schedule: 70%.

Program: Active Electronically Scanned Array radar (upgrade for F/A-18 
E/F fighter/attack aircraft);
Percent growth in estimated development cost[A]: 14%;
Percent growth in estimated development schedule: 1%.

Program: Global Hawk unmanned aerial vehicle;
Percent growth in estimated development cost[A]: 166%;
Percent growth in estimated development schedule: Undetermined.

Program: Joint Strike Fighter;
Percent growth in estimated development cost[A]: 30%;
Percent growth in estimated development schedule: 23%.

Program: UH-60M helicopter upgrade;
Percent growth in estimated development cost[A]: 151%;
Percent growth in estimated development schedule: 25%.

Program: C-130 Avionics Modernization Program;
Percent growth in estimated development cost[A]: 122%;
Percent growth in estimated development schedule: Undetermined.

Program: C-5 Reliability Enhancement and Re-engining Program;
Percent growth in estimated development cost[A]: 0%;
Percent growth in estimated development schedule: 25%.

Program: Joint Tactical Radio System Cluster 1;
Percent growth in estimated development cost[A]: 31%;
Percent growth in estimated development schedule: 44%.

Program: Joint Tactical Radio System Waveform;
Percent growth in estimated development cost[A]: 44%;
Percent growth in estimated development schedule: Undetermined.

Program: Advanced Anti-radiation Guided Missile;
Percent growth in estimated development cost[A]: 7%;
Percent growth in estimated development schedule: 0%.

Program: Multi-Platform Radar Technology Insertion Program;
Percent growth in estimated development cost[A]: 0%;
Percent growth in estimated development schedule: Undetermined.

Program: Future Combat System;
Percent growth in estimated development cost[A]: 48%;
Percent growth in estimated development schedule: 53%.

Program: E-2 Advanced Hawkeye;
Percent growth in estimated development cost[A]: 5%;
Percent growth in estimated development schedule: 0%.

Program: Warfighter Information Network-Tactical;
Percent growth in estimated development cost[A]: 0%;
Percent growth in estimated development schedule: 0%.

Program: Small Diameter Bomb;
Percent growth in estimated development cost[A]: 0%;
Percent growth in estimated development schedule: 0%.

Program: EA-18G;
Percent growth in estimated development cost[A]: 7%;
Percent growth in estimated development schedule: 0%.

Program: Joint Tactical Radio System Cluster 5;
Percent growth in estimated development cost[A]: 0%;
Percent growth in estimated development schedule: 2%.

Program: Multi-Mission Maritime Aircraft;
Percent growth in estimated development cost[A]: 0%;
Percent growth in estimated development schedule: 0%.

Program: Standard Missile-6 Extended Range Active Missile Block 1;
Percent growth in estimated development cost[A]: 0%;
Percent growth in estimated development schedule: 0%.

Program: Aerial Common Sensor;
Percent growth in estimated development cost[A]: 45%;
Percent growth in estimated development schedule: 36%.

Program: B-2 Radar Modernization Program;
Percent growth in estimated development cost[A]: 0%;
Percent growth in estimated development schedule: 0%.

Program: Patriot/Medium Extended Air Defense System Combined Aggregate 
Program (fire unit);
Percent growth in estimated development cost[A]: 0%;
Percent growth in estimated development schedule: 0%.

Program: Mission Planning System;
Percent growth in estimated development cost[A]: 0%;
Percent growth in estimated development schedule: 0%.

Sources: DOD (data); GAO (analysis and presentation).

[A] Cost growth is expressed as the percent change in program 
development cost estimates in fiscal year 2005 dollars.

[End of table]

The Army's Future Combat System is a case in point. Less than 3 years 
after program initiation and with $4.6 billion invested, the Army has 
already increased its development cost estimate $8.9 billion or 48 
percent and delayed delivery of initial capability by 4 years over the 
original business case. Similarly, just over 1 year after initiating 
development of the Aerial Common Sensor aircraft, the Army has reported 
that severe weight and design problems discovered during development 
have stopped work on the program. As a result, program officials are 
anticipating at least a 45 percent cost increase and a delay of 2 years 
in delivering an initial capability to the warfighter. These two Army 
programs are not the only ones experiencing problems. Table 2 contains 
cost and schedule data for 6 of the 10 largest development programs 
initiated under the revised policy, including the Future Combat System 
and Aerial Common Sensor. As the table illustrates there are several 
programs experiencing large cost increases and schedule delays.

Table 2: Cost and Schedule Outcomes for 6 of the 10 Largest Development 
Programs Sorted by Percent of System Development Remaining:

Programs: Aerial Common Sensor;
Percent development cost growth: 45%;
Delay in delivery of initial capability in months: 24;
Percent of development remaining: 85%.

Programs: Future Combat System;
Percent development cost growth: 48%;
Delay in delivery of initial capability in months: 48;
Percent of development remaining: 78%.

Programs: Joint Strike Fighter;
Percent development cost growth: 30%;
Delay in delivery of initial capability in months: 23;
Percent of development remaining: 60%.

Programs: Expeditionary Fighting Vehicle;
Percent development cost growth: 61%;
Delay in delivery of initial capability in months: 48;
Percent of development remaining: 49%.

Programs: C-130 Avionics Modernization Program;
Percent development cost growth: 122%;
Delay in delivery of initial capability in months: Delays anticipated 
due to program restructure;
Percent of development remaining: Undetermined due to program 
restructure.

Programs: Global Hawk (RQ-4B);
Percent development cost growth: 166%;
Delay in delivery of initial capability in months: Delays anticipated 
due to program restructure;
Percent of development remaining: Undetermined due to program 
restructure.

Sources: DOD (data); GAO (analysis and presentation).

[End of table]

A good measure of acquisition performance is return on investment as 
expressed in acquisition program unit cost because unit cost represents 
the value DOD is getting for its acquisition dollars invested in a 
certain program. The programs listed in table 2 will not achieve the 
return on investment that DOD anticipated when they began development. 
In the case of Joint Strike Fighter, for example, DOD initially 
intended to purchase 2,866 aircraft at an acquisition program unit cost 
of about $66 million. The Navy has reduced the number of Joint Strike 
Fighter aircraft it plans to buy; technology and design problems 
encountered during development have led to the significant cost growth. 
As a result, the acquisition program unit cost is now about $84 
million, an increase of 27 percent. We recently reported that the risk 
of even greater increases is likely because flight testing has not yet 
started and the acquisition strategy involves substantial overlap of 
development and production. Similar problems have led to increases in 
the Future Combat System program. At program initiation, the Army 
anticipated that each of 15 units would cost about $5.5 billion to 
develop and deliver. Since that time, instability in the program's 
technologies and requirements have led to significant cost increases, 
leading to a 54 percent increase in acquisition program unit cost, now 
estimated to be $8.5 billion.

Regarding all 23 development programs, DOD leaders originally planned 
to invest a total of about $83 billion (fiscal year 2005 dollars) for 
system development and anticipated delivering an initial operational 
capability to the warfighter in 77 months on average. However, 
development costs have grown and delivery schedules have been delayed 
significantly. DOD now expects to invest over $106 billion in those 
same programs, an increase of over $23 billion or 28 percent. The 
delivery of initial capability to the warfighter is expected to take an 
average of 88 months or nearly 1 year longer than originally planned. 
Figure 3 shows changes in these business case elements for these 
programs in the short time since their initiation.

Figure 3: Cost and Schedule Growth under DOD's Revised Policy:

[See PDF for image]

Sources: DOD (data); GAO (analysis and presentation).

[End of figure]

DOD Is Not Effectively Implementing the Policy's Knowledge-Based, 
Evolutionary Approach:

DOD is not effectively implementing the knowledge-based process and 
evolutionary approach emphasized in its acquisition policy. While the 
policy outlines a specific knowledge-based process of concept 
refinement and technology development[Footnote 3] to help ensure a 
sound business case is developed before committing to a new development 
program, almost 80 percent of the programs we reviewed were permitted 
to bypass this process. Furthermore, the policy emphasizes the need to 
mature all critical technologies before starting system development and 
to demonstrate that the product's design is mature before beginning 
system demonstration. However, nearly three-fourths of the programs 
reported having immature critical technologies when they received 
approval to start development, and at least half of the programs had 
not achieved design maturity before holding their design review and 
gaining approval to enter the system demonstration phase of 
development. The policy also emphasizes the use of an evolutionary 
product development approach, yet program officials continue to 
structure major acquisition programs to achieve large advances in 
capability within a single step development program. This strategy has 
historically resulted in poor cost and schedule outcomes.

Knowledge-Based Process Not Enforced:

DOD decision makers continue to approve programs for system development 
that have not followed key elements of the policy's suggested knowledge-
based process. The policy requires program managers to provide senior 
decision makers with knowledge about key aspects of a system at 
critical investment points in the acquisition process. Our prior 
reviews have identified those critical points as the start of system 
development or program start (referred to as Milestone B in the DOD 
acquisition policy), design readiness review separating system 
integration and system demonstration, and production commitment 
(Milestone C in the DOD acquisition policy). The most important point 
occurs at program start, when system development begins. DOD 
acquisition guidance emphasizes the importance of the acquisition 
phases preceding program start, noting that the decisions made during 
those phases--concept refinement and technology development--generally 
define the nature of an entire acquisition program.

Acquisition Officials Are Not Effectively Using Early Processes to 
Develop Executable Business Cases:

Acquisition officials continue to begin system development without 
following early processes for developing executable business cases. A 
business case should provide demonstrated evidence that (1) the 
warfighter's needs are real and necessary and that they can best be met 
with the chosen concept and (2) the chosen concept can be developed and 
produced within existing resources--including technologies, design 
knowledge, funding, and the time to deliver the product when it is 
needed. Establishing a business case calls for a realistic assessment 
of risks and costs; doing otherwise undermines the intent of the 
business case and invites failure. This process requires the user and 
developer to negotiate whatever trade-offs are needed to achieve a 
match between the user's requirements and the developer's resources 
before system development begins.

The revised policy and associated guidance emphasize the importance of 
following a sound process of systems engineering[Footnote 4] and 
decision making prior to initiating a system development program. The 
process established in the policy consists of two phases, concept 
refinement and technology development, and a major decision review 
called Milestone A, which if rigorously followed, would provide 
acquisition officials with an opportunity to assess whether program 
officials had the knowledge needed to develop an executable business 
case. However, almost 80 percent of the programs we reviewed began 
system development without holding any prior decision review. Senior 
officials with the Office of the Secretary of Defense confirmed that 
this is a common practice among defense acquisition programs. This 
practice eliminates a key opportunity for decision makers to assess 
early product knowledge needed to establish a business case that is 
based on realistic cost, schedule, and performance expectations.

Although program officials conduct analysis before starting a 
development program, they do not consistently follow a process to 
capture the critical knowledge needed to produce executable business 
cases, as evidenced by the poor outcomes current programs are 
experiencing. Officials with the Office of the Secretary of Defense 
recognized this lack of rigor and discipline in acquisition process, 
and in February 2004, the Under Secretary of Defense (Acquisition, 
Technology and Logistics) issued a department-wide policy memorandum 
directing acquisition officials to place greater emphasis on systems 
engineering when planning and managing acquisition programs. The policy 
requires programs to develop a systems engineering plan that describes 
the programs' overall technical approach, including processes, 
resources, metrics, and applicable performance incentives. Although 
DOD's systems engineering initiative has the potential to improve 
program performance, officials have found that the preliminary results 
are mixed. Early analysis shows that implementation is inconsistent 
while program officials learn to develop and implement systems- 
engineering plans.

Programs Continue to Enter System Development with Immature 
Technologies:

DOD decision makers continue to permit programs to enter system 
development before critical technologies are mature. Our review of 
technology readiness assessments and acquisition decision memorandums 
for our nine case study programs found that seven of the nine programs 
were approved to begin development even though program officials 
reported levels of knowledge below the criteria suggested in the policy 
and associated guidance, specifically in the area of technology 
maturity.[Footnote 5] Those seven programs are not isolated cases. As 
illustrated in Figure 4, 13 of the programs (nearly three-fourths) that 
received approval to enter system development under the revised policy 
did so with less than 100 percent of their critical technologies mature 
to the level specified by DOD. Only 2 of those programs had more than 
75 percent of their technologies mature when they began (see appendix 
III for technology maturity data for each program).

Figure 4: Comparison of Programs with Mature versus Immature 
Technologies at Start of System Development:

[See PDF for image]

Sources: DOD (data); GAO (analysis and presentation).

Note: This figure depicts technology maturity status for 18 of the 23 
programs in our review. Technology maturity data was not available for 
the other 5 programs.

[End of figure]

Even though acquisition policy states that technologies shall be mature 
before beginning system development, the practice of accepting high 
levels of technology risk at program start continues to be the norm and 
not the exception. An official with Office of the Secretary of Defense 
responsible for reviewing and validating program assessments of 
technology maturity informed us that the office generally views 
immature critical technologies at the beginning of development as an 
acceptable risk as long as program officials can show that they have a 
plan to mature the technologies by the time the program reaches its 
design readiness review, which requires additional investments to move 
a program from system integration into system demonstration. Therefore, 
risk management plans are consistently viewed as acceptable substitutes 
for demonstrated knowledge.

Programs Continue Past Design Reviews before Design Maturity is 
Demonstrated:

In addition to emphasizing the importance of capturing technology 
knowledge before starting system development, DOD's policy also 
highlights the importance of demonstrating design maturity before 
moving from the integration phase of system development into system 
demonstration and initial manufacturing. The policy establishes a 
design readiness review between the two phases to determine whether a 
product's design is mature and stable and whether the product is ready 
to move ahead. While DOD's policy does not require programs to 
demonstrate any specific level of design maturity, our past work has 
found that a key indicator of design maturity is the completion of 90 
percent of the system's engineering drawings. We found that defense 
programs that moved forward with lower levels of design maturity, as 
indicated by drawing completion, encountered costly design changes and 
parts shortages that, in turn, caused labor inefficiencies, schedule 
delays, and quality problems. Consequently, those programs required 
significant increases in resources--time and money--over what was 
estimated at the point each program entered the system demonstration 
phase.

We analyzed engineering drawing completion data for 8 programs 
initiated under the revised policy that have held a design 
review,[Footnote 6] and found that more than half of those programs had 
not completed 90 percent of their design drawings before they received 
approval to enter the system demonstration phase of development. We 
also analyzed drawing-release data for three programs that have not yet 
held their design review but have projected the number of drawings 
officials anticipate will be completed when their reviews are held. 
Based on projections provided by program officials, 2 of those 3 
programs are expected to have less than 55 percent of their drawings 
complete before they seek approval to begin system demonstration and 
initial manufacturing.

Table 3: Assessment of Program Design Maturity:

Program: Joint Tactical Radio System Cluster 1;
Percentage of design drawings complete at design review: 28%.

Program: Global Hawk unmanned aerial vehicle;
Percentage of design drawings complete at design review: 33%.

Program: Active Electronically Scanned Array radar (upgrade for F/A-18 
E/F fighter/attack aircraft);
Percentage of design drawings complete at design review: 59%.

Program: Expeditionary Fighting Vehicle;
Percentage of design drawings complete at design review: 81%.

Program: B-2 Radar Modernization Program;
Percentage of design drawings complete at design review: 84%.

Program: E-2 Advanced Hawkeye;
Percentage of design drawings complete at design review: 90%.

Program: EA-18G;
Percentage of design drawings complete at design review: 97%.

Program: C-5 Reliability Enhancement and Re-engining Program;
Percentage of design drawings complete at design review: 98%.

Program: Multi-Platform Radar Technology Insertion Program;
Percentage of design drawings complete at design review: 100%[A].

Program: Joint Strike Fighter;
Percentage of design drawings complete at design review: 52%[A].

Program: Aerial Common Sensor;
Percentage of design drawings complete at design review: 39%[A].

Sources: DOD (data); GAO (analysis and presentation).

[A] Program office projections.

[End of table]

Evolutionary Acquisition Is Not Being Used:

Despite the revised policy's guidance that capabilities should be 
developed and delivered in individually defined and separately managed 
increments, a majority of major weapon acquisition programs we assessed 
continue to be structured to achieve revolutionary increases in 
capability within one development program. According to the policy, the 
objective of an evolutionary approach is to balance needs and available 
capability with resources and put capability into the hands of the user 
quickly. The policy states that the success of the strategy depends on 
consistent and continuous definition of requirements and the maturation 
of technologies that lead to disciplined development and production of 
systems that provide increasing capability. In this approach, 
requirements that cannot be satisfied within these limits as well as 
available financial resources must wait for future generations of the 
product and be managed as separate system development programs with 
separate milestones, costs, and schedules. In our case studies of nine 
acquisition programs initiated under the revised policy, we found only 
one program--the Small Diameter Bomb--that satisfied all of the 
criteria of an evolutionary approach. In five case studies, we found 
that program officials had claimed that their programs were 
evolutionary, yet our evidence shows they were not evolutionary in 
practice;[Footnote 7] and in three cases, program officials chose not 
to use evolutionary acquisition from the outset. Table 3 summarizes our 
assessment of the nine case studies.

Table 4: Assessment of Program Acquisition Strategies for GAO's Nine 
Case Studies:

Programs in GAO's case study: Future Combat System;
Claim to be evolutionary?: Yes;
Meet evolutionary criteria?: No;
Greater than 30% cost growth or more than 1-year schedule slip: Yes.

Programs in GAO's case study: Global Hawk (RQ-4B);
Claim to be evolutionary?: Yes;
Meet evolutionary criteria?: No;
Greater than 30% cost growth or more than 1-year schedule slip: Yes.

Programs in GAO's case study: Joint Strike Fighter;
Claim to be evolutionary?: Yes;
Meet evolutionary criteria?: No;
Greater than 30% cost growth or more than 1-year schedule slip: Yes.

Programs in GAO's case study: Aerial Common Sensor;
Claim to be evolutionary?: Yes;
Meet evolutionary criteria?: No;
Greater than 30% cost growth or more than 1-year schedule slip: Yes.

Programs in GAO's case study: Multi-Mission Maritime Aircraft;
Claim to be evolutionary?: Yes;
Meet evolutionary criteria?: No;
Greater than 30% cost growth or more than 1-year schedule slip: No.

Programs in GAO's case study: Small Diameter Bomb;
Claim to be evolutionary?: Yes;
Meet evolutionary criteria?: Yes;
Greater than 30% cost growth or more than 1-year schedule slip: No.

Programs in GAO's case study: E-2 Advanced Hawkeye;
Claim to be evolutionary?: No;
Meet evolutionary criteria?: No;
Greater than 30% cost growth or more than 1-year schedule slip: No.

Programs in GAO's case study: Expeditionary Fighting Vehicle;
Claim to be evolutionary?: No;
Meet evolutionary criteria?: No;
Greater than 30% cost growth or more than 1-year schedule slip: Yes.

Programs in GAO's case study: Multiplatform Radar Technology Insertion 
Program;
Claim to be evolutionary?: No;
Meet evolutionary criteria?: No;
Greater than 30% cost growth or more than 1-year schedule slip: No.

Sources: DOD (data); GAO (analysis and presentation).

[End of table]

Specific Criteria Are Needed to Ensure Disciplined and Transparent 
Investment Decisions:

The revised acquisition policy does not contain effective controls that 
require the demonstration of product knowledge measured against 
specific criteria to ensure that acquisition officials make 
disciplined, transparent, and knowledge-based investment decisions. The 
lack of specific required criteria creates an environment in which 
unknowns about technology, design, and manufacturing processes are 
acceptable. Decision makers and program officials are left with no 
objective measures against which to gauge a program's level of 
knowledge, making accountability difficult. In the absence of criteria, 
transparency in acquisition decisions is essential to ensuring 
accountability, but key decision documents do not provide sufficient 
information about major decisions. DOD believes that acquisition 
decision memorandums, used to document program decisions, provide 
adequate transparency. However, the decision memorandums we reviewed 
did not contain an explanation of the decision maker's rationale and 
rarely identify remaining risks, especially as they relate to the key 
knowledge standards emphasized in the policy. Further, the timeliness, 
accessibility, and depth, of the data contained in the Selected 
Acquisition Reports, DOD's primary means of providing Congress with a 
status report of program performance, inhibits the reports' usefulness 
as a management and oversight tool.

In November 2003, we reported that the revised acquisition policy 
lacked many of the controls that leading commercial companies rely on 
to attain an acceptable level of knowledge before making additional 
significant investments.[Footnote 8] Controls are considered effective 
if they are backed by specific criteria and if decision makers are 
required to consider the resulting data before deciding to advance a 
program to the next level. Controls used by leading companies help 
decision makers gauge progress in meeting cost, schedule, and 
performance goals and hold program managers accountable for capturing 
relevant product knowledge to inform key investment decisions. The 
controls we have articulated as best practices used by successful 
commercial product developers are listed below in table 5.

Table 5: Types of Controls Considered Best Practices for Successful 
Product Development:

Program start (Milestone B): Start of product development: Demonstrate 
technologies to high readiness levels.

Program start (Milestone B): Start of product development: Ensure that 
requirements for the product are informed by the systems engineering 
process.

Program start (Milestone B): Start of product development: Establish 
cost and schedule estimates for product on the basis of knowledge from 
preliminary design using system engineering tools.

Program start (Milestone B): Start of product development: Conduct 
decision review for program start.

Design Readiness Review: Beginning of System demonstration: Complete 90 
percent of design drawings.

Design Readiness Review: Beginning of System demonstration: Complete 
subsystem and system design reviews.

Design Readiness Review: Beginning of System demonstration: Demonstrate 
with prototype that design meets requirements.

Design Readiness Review: Beginning of System demonstration: Obtain 
stakeholders' concurrence that drawings are complete and producible.

Design Readiness Review: Beginning of System demonstration: Complete 
the failure modes and effects analysis.

Design Readiness Review: Beginning of System demonstration: Identify 
key system characteristics.

Design Readiness Review: Beginning of System demonstration: Identify 
critical manufacturing processes.

Design Readiness Review: Beginning of System demonstration: Establish 
reliability targets and growth plan on the basis of demonstrated 
reliability rates of components and subsystems.

Design Readiness Review: Beginning of System demonstration: Conduct 
decision review to enter system demonstration.

Production Commitment (Milestone C): Initiation of low-rate production: 
Demonstrate manufacturing processes.

Production Commitment (Milestone C): Initiation of low-rate production: 
Build production-representative prototypes.

Production Commitment (Milestone C): Initiation of low-rate production: 
Test production-representative prototypes to achieve reliability goal.

Production Commitment (Milestone C): Initiation of low-rate production: 
Test production-representative prototypes to demonstrate product in 
operational environment.

Production Commitment (Milestone C): Initiation of low-rate production: 
Collect statistical process control data.

Production Commitment (Milestone C): Initiation of low-rate production: 
Demonstrate that critical processes are capable and in statistical 
control.

Production Commitment (Milestone C): Initiation of low-rate production: 
Conduct decision review to begin production.

Sources: GAO (analysis and presentation).

[End of table]

Some senior officials with the Office of the Secretary of Defense 
believe that the effective use of controls in DOD's policy and the 
establishment of more specific criteria for decision making would 
improve program outcomes. They note that specific criteria need to be 
established and that programs need to be held accountable to those 
criteria before being permitted to proceed into the next phase. They 
also note that the criteria for moving an acquisition effort from one 
phase of the process to the next, primarily documented in acquisition 
decision memorandums as exit criteria, are not typically specific and 
often do not relate to the key knowledge-based criteria suggested in 
the policy.

We found this to be true for our nine case study programs. We reviewed 
acquisition decision memorandums in our case studies and determined 
that they were not useful in explaining the decision maker's rationale 
and in almost all of the cases they did not address the key knowledge 
criteria suggested in the acquisition policy. In most instances, the 
decision maker simply noted that the program being assessed was ready 
to proceed into system development, but did not provide an explanation 
of the rationale for the decision. Senior officials with the Office of 
the Secretary of Defense told us that they agree that a better 
explanation of the decision maker's rationale, specifically in 
instances where the knowledge criteria are not fully met, would provide 
transparency and ultimately allow for a more accountable decision- 
making process. The following two examples illustrate how decision 
documentation is lacking:

* The Future Combat System program received approval to enter system 
development and demonstration in 2003, with 19 percent of its critical 
technologies mature, well below the policy's standard. The acquisition 
decision memorandum supporting this decision did not provide the 
rationale for approving the system with such a large number of immature 
critical technologies. The memo did direct an updated review of the 
decision 18 months later and that the program "remain flexible and open 
to accommodate trades in the system architecture and in the individual 
systems' designs."

* The Joint Strike Fighter program was approved to enter system 
development in 2001. The acquisition decision memorandum did not 
address the fact that 75 percent of the program's critical technologies 
were not mature to the policy's standard. The memorandum did 
acknowledge that the program's requirements could be changed or 
modified, noting that further refinements in the requirements should be 
explored as a potential way to reduce program costs. However, the 
memorandum did not explain why the decision maker determined that the 
program should enter development without achieving the technology and 
requirements knowledge emphasized in the policy.

The acquisition decision memorandums for most of the other programs we 
reviewed did not specifically address critical gaps in knowledge, nor 
did they effectively explain the decision makers' rationale for deeming 
those programs ready to begin system development. In memos where we 
found a reference to key knowledge principles, such as technology 
maturity, the decision makers acknowledged that more effort was needed 
to meet the policy's suggested criteria but considered the risk 
acceptable to begin development. These memos did not explain why risks 
were considered acceptable. For example, the Navy's Multi-Mission 
Maritime Aircraft program had none of its critical technologies mature 
at program initiation. The decision maker acknowledged the need to 
further mature the critical technologies but approved the program to 
enter development. Instead of holding the program to the policy's 
criteria for entering development, the decision maker simply directed 
the Navy to work with the Office of the Secretary of Defense to 
implement risk mitigation and technology maturation plans during the 
integration phase of system development.

In addition to the lack of transparency provided through acquisition 
decision memoranda, we also found that the data presented to Congress 
in DOD's Selected Acquisition Reports (SARs) provided only limited 
usefulness as an oversight tool. Since 1969, SARs have been the primary 
means by which DOD reports the status of major weapon system 
acquisitions to Congress. SARs are reports that are expected to contain 
information on the cost, schedule, and performance of major weapon 
systems in comparison with baseline values established at program 
start, full-scale development, and production decision points. Our 
analysis, as well as a previous GAO review,[Footnote 9] of current and 
historical SAR data found that the timeliness, accessibility, and depth 
of the data contained in the reports limits their usefulness as an 
oversight tool. Our prior review noted that a number of opportunities 
exist for DOD to give Congress more complete information on the 
performance of major defense acquisition programs. DOD agreed that SAR 
data could be improved to make it more useful to Congress.

Conclusions:

Failing to consistently implement the knowledge-based process and 
evolutionary principles emphasized in the revised acquisition policy-- 
coupled with a lack of specific criteria for making key investment 
decisions--are keeping DOD on its historical path of poor cost and 
schedule outcomes. Most programs are incurring the same scope of cost 
overruns and schedule delays as programs managed under prior DOD 
policies. More consistent use of the early acquisition processes would 
improve the quality and viability of program business cases by ensuring 
they are founded on knowledge obtained from rigorous and disciplined 
analysis. The initiative by Office of the Secretary of Defense to 
reinstitute the use of systems engineering is a step in the right 
direction. However, in order for this initiative to be effective DOD 
must establish and enforce specific criteria at key decision points. 
Our past work has identified and recommended criteria and controls that 
should be consistently applied at major decision points. The 
enforcement of these criteria is critical to ensuring that programs 
have the knowledge necessary to successfully move forward through the 
acquisition process. DOD officials have acknowledged the advantages of 
using knowledge-based criteria and controls, but believe the policy 
already includes enough controls to achieve effective program results. 
However, without enforceable criteria, defense officials are challenged 
to determine whether adequate knowledge has been obtained for investing 
taxpayer dollars. The lack of enforceable criteria also makes it 
difficult to hold defense officials accountable for their decisions.

Recommendations for Executive Action:

DOD must ensure that appropriate knowledge is captured and used at 
critical junctures to make decisions about moving a program forward and 
investing more money. We recommend that the Secretary of Defense 
require program officials to demonstrate that they have captured 
appropriate knowledge at three key points--program start, design review 
for transitioning from system integration to system demonstration, and 
production commitment--as a condition for investing resources. At a 
minimum those controls should require program officials to demonstrate 
that they have achieved a level of knowledge that meets or exceeds the 
following criteria at each respective decision point:

* Program start (Milestone B): Start of product development:

- Demonstrate technologies to high readiness levels:

- Ensure that requirements for the product are informed by the systems- 
engineering process:

- Establish cost and schedule estimates for product on the basis of 
knowledge from preliminary design using system engineering tools:

- Conduct decision review for program start:

* Design readiness review: Beginning of system demonstration:

- Complete 90 percent of design drawings:

- Complete subsystem and system design reviews:

- Demonstrate with prototype that design meets requirements:

- Obtain stakeholders' concurrence that drawings are complete and 
producible:

- Complete the failure modes and effects analysis:

- Identify key system characteristics:

- Identify critical manufacturing processes:

- Establish reliability targets and growth plan on the basis of 
demonstrated reliability rates of components and subsystems:

- Conduct decision review to enter system demonstration:

* Production commitment (Milestone C): Initiation of low-rate 
production:

- Demonstrate manufacturing processes:

- Build production-representative prototypes:

- Test production-representative prototypes to achieve reliability goal:

- Test production-representative prototypes to demonstrate product in 
operational environment:

- Collect statistical process control data:

- Demonstrate that critical processes are capable and in statistical 
control:

- Conduct decision review to begin production:

Furthermore, to ensure that major decisions are transparent and that 
program officials and decision makers are held accountable, we 
recommend that the Secretary of Defense require decision makers to 
include written rationale for each major decision in acquisition 
decision documentation. The rationale should address the key knowledge- 
based criteria appropriate for milestone decisions, explain why a 
program's level of knowledge in each area was deemed acceptable if 
criteria have not been met and provide a plan for achieving the 
knowledge necessary to meet criteria within a given time frame.

Agency Comments and Our Evaluation:

DOD provided written comments on a draft of this report. The comments 
appear in appendix II.

DOD partially concurred with our recommendation that the Secretary of 
Defense should establish specific controls to insure that program 
officials demonstrate that they have captured a level of knowledge that 
meets or exceeds specific criteria at three key points in the 
acquisition process: program start, design readiness review, and 
production commitment. DOD agreed that knowledge-based decision making 
is consistent with sound business practice and stated that it would 
continue to develop policy that reflects a knowledge-based approach and 
improves acquisition outcomes. DOD noted that it would consider our 
recommendations as it reassesses the DOD acquisition business model and 
the knowledge required at each decision point. We believe that DOD's 
plan to reassess its business model provides a good opportunity to 
establish the controls and specific criteria recommended in this 
report. Therefore, we are retaining our recommendation that the 
Secretary of Defense should establish controls to insure that program 
officials demonstrate that they have captured a level of knowledge that 
meets or exceeds specific criteria at three key points in the 
acquisition process.

DOD also partially concurred with our recommendation that the Secretary 
of Defense require decision makers to provide written rationale in 
acquisition decision documentation for each major decision. DOD agreed 
that acquisition decisions should be documented, decision makers should 
be held accountable, and that they should provide the rationale for 
their decisions. DOD believes that the implementation of Section 801 of 
the National Defense Authorization Act for FY 2006 reinforces these 
processes. The act calls for the decision maker to certify that the 
program meets certain requirements, such as technology maturity, prior 
to starting a new development program at Milestone B. However, the act 
is focused on the decision to start a development program and does not 
identify specific criteria for programs to be measured against at 
design readiness review or production commitment. We believe our 
recommendation adds transparency and accountability to the process 
because it requires the decision maker to provide the rationale for a 
decision to allow a program to move forward, not only at Milestone B 
but at other key decision points as well. Therefore, we are retaining 
our recommendation that the Secretary of Defense require decision 
makers to provide written rationale for each major decision in 
acquisition decision documentation.

We are sending copies of this report to the Secretary of Defense; the 
Secretaries of the Air Force, Army, and Navy; and the Director of the 
Office of Management and Budget. We will provide copies to others on 
request. This report will also be available at no charge on GAO's Web 
site at [Hyperlink=http://www.gao.gov].

If you have any questions about this report or need additional 
information, please call me at (202) 512-4841 (sullivanm@gao.gov). 
Contact points for the offices of Congressional Relations and Public 
Affairs are located on the last page of this report. Key contributors 
to this report were Michael Hazard, Assistant Director; Lily Chin; Ryan 
Consaul; Christopher DePerro; Travis Masters; and Adam Vodraska.

Signed by:

Michael J. Sullivan:
Director, Acquisition and Sourcing Management:

[End of section]

Appendix I: Scope and Methodology:

To assess the impact of DOD's revised acquisition policy, we analyzed 
cost and schedule data for 23 major defense acquisition programs that 
were approved to begin system development under the revised policy. We 
did not assess space, missile defense, or ship programs. We collected 
our data from Selected Acquisition Reports, presidential budget 
documents, ongoing GAO work, and pertinent program officials. We 
utilized previous GAO reports related to defense acquisition policies 
and worked with knowledgeable GAO staff to ensure the use of current, 
accurate data. We also analyzed more than 150 annual Selected 
Acquisition Reports covering a 36-year period from 1969 to 2005, to 
determine historical trends related to outcomes of acquisition policy 
implementation.

We assessed whether the revised policy's knowledge-based, evolutionary 
acquisition principles were being effectively implemented by conducting 
9 case study reviews and analyzing design maturity data for 11 programs 
that have made engineering-drawing data available to GAO. Our case 
study programs were the Aerial Common Sensor, Multi-Platform Radar 
Technology Insertion Program, Global Hawk Unmanned Aerial Vehicle, 
Small Diameter Bomb, Future Combat System, Joint Strike Fighter, 
Expeditionary Fighting Vehicle, Multi-Mission Maritime Aircraft, and 
the E-2 Advanced Hawkeye. We interacted directly with numerous program 
officials to seek input on current developments with their programs. We 
studied program documents to assess how well programs understand and 
are implementing the revised acquisition policy. We also analyzed 
drawing release data for those programs that have either passed their 
design review or have provided GAO with estimated drawing release data 
for a future design review to assess design maturity. In several cases, 
we asked that program offices verify information in these various 
documents.

We also reviewed Department of Defense (DOD) Directive 5000.1, DOD 
Instruction 5000.2, and the Defense Acquisition Guidebook. In addition 
we examined each of the military services' policy directives and 
guidance, DOD memorandums to include policy intent and DOD expectations 
regarding policy implementation as well as Joint Capabilities 
Integration and Development System documents. We interviewed relevant 
officials in Washington, D.C., from the Office of the Director, Defense 
Research and Engineering, the Joint Staff, the Office of the Secretary 
of Defense, and Army, Navy, and Air Force acquisition policy staff in 
order to better understand the content of these documents and the 
intent of DOD's policy.

We conducted our review from May 2005 to February 2006 in accordance 
with generally accepted government auditing standards.

[End of section]

Appendix II: Comments from the Department of Defense:

Acquisition Technology and Logistics:

Office Of The Under Secretary Of Defense:
9000 Defense Pentagon Washington, DC 20301-9000:

April 13 2006:

DPAP/PAIC:

Mr. Michael Sullivan:
Director, Acquisition and Sourcing Management:
U.S. Government Accountability Office:
441 G Street, N.W.
Washington, D.C. 20548:

Dear Mr. Sullivan:

This is the Department of Defense response to the GAO draft report, 
"DEFENSE ACQUISITIONS: Major Weapon Systems Continue to Experience Cost 
and Schedule Problems under DOD's Revised Policy" (GAO-06-368/GAO Code 
120448) DOD Acquisition Policy Implementation. I concur with the 
recommendation in the report. Additional comment is attached.

Sincerely,

Signed by:

Shay Assad
Director, Defense Procurement and Acquisition Policy

Attachment: As stated:

GAO DRAFT REPORT - DATED MARCH 7, 2006:
GAO CODE 120448 /GAO-06-368:

"DEFENSE ACQUISITIONS: Major Weapon Systems Continue to Experience Cost 
and Schedule Problems under DOD's Revised Policy"

DEPARTMENT OF DEFENSE COMMENT ON THE RECOMMENDATION:

GAO RECOMMENDATION:

DOD must ensure that appropriate knowledge is captured and used at 
critical junctures to make decisions about moving a program forward and 
investing more money. We recommend that the Secretary of Defense 
require program officials to demonstrate that they have captured 
appropriate knowledge at three key points - program start, design 
review for transitioning from system integration to system 
demonstration, and production commitment - as a condition for investing 
resources. At a minimum those controls should require program officials 
to demonstrate that they have achieved a level of knowledge that meets 
or exceeds the following criteria at each respective decision point:

Program start (Milestone B): Start of product development

* Demonstrate technologies to high readiness levels:

* Ensure that requirements for the product are informed by the systems 
engineering process:

* Establish cost and schedule estimates for product on the basis of 
knowledge from preliminary design using system engineering tools:

* Conduct decision review for program start:

Design readiness review: Beginning of system demonstration 

* Complete 90 percent of design drawings:

* Complete subsystem and system design reviews:

* Demonstrate with prototype that design meets requirements:

* Obtain stakeholders' concurrence that drawings are complete and 
producible:

* Complete the failure modes and effects analysis:

* Identify key system characteristics:

* Identify critical manufacturing processes:

* Establish reliability targets and growth plan on the basis of 
demonstrated reliability rates of components and subsystems:

* Conduct decision review to enter system demonstration:

Production commitment (Milestone C): Initiation of low-rate production 

* Demonstrate manufacturing processes:

* Build production-representative prototypes:

* Test production-representative prototypes to achieve reliability goal:

* Test production-representative prototypes to demonstrate product in 
operational environment:

* Collect statistical process control data:

* Demonstrate that critical processes are capable and in statistical 
control:

* Conduct decision review to begin production:

Furthermore, to ensure that major decisions are transparent and that 
program officials and decision makers are held accountable, we 
recommend that the Secretary of Defense require decision makers to 
include written rationale for each major decision in acquisition 
decision documentation. The rationale should address the key knowledge- 
based criteria appropriate for milestone decisions, explain why a 
program's level of knowledge in each area was deemed acceptable if 
criteria have not been met, and provide a plan for achieving the 
knowledge necessary to meet criteria within a given time frame.

DOD RESPONSE: Partial Concurrence:

Partially Concur. The Department of Defense agrees that knowledge-based 
decision making is consistent with sound business practice. 
Consequently, we will continue to develop policy that reflects that 
approach and improves acquisition outcomes. In keeping with that 
objective and in association with the results of the recently completed 
Quadrennial Defense Review, we will soon be reassessing the DOD 
acquisition business model and the knowledge required at each decision 
point. In that vein, we will consider the GAO proposals during that 
process. We also agree that acquisition decisions should be documented, 
that decision makers should be held accountable, and that they should 
provide the rationale for their decisions-a process reinforced through 
implementation of Section 801 of the National Defense Authorization Act 
for FY 2006.

[End of section]

Appendix III: Program Data for 23 Programs Initiated under DOD's 
Revised Acquisition Policy (as of December 2005):

Program: Expeditionary Fighting Vehicle;
Program start: 12/2000;
Formal Milestone I[A] or Milestone A decision review?: Yes;
Percent technology mature (TRL 6) at program start: 80%;
Percent design drawings complete at design review: 81%;
Percent growth in estimated development cost[C]: 61%;
Percent growth in estimated development schedule: 70%.

Program: Active Electronically Scanned Array radar (upgrade for F/A-18 
E/F fighter/attack aircraft);
Program start: 12/2000;
Formal Milestone I[A] or Milestone A decision review?: No;
Percent technology mature (TRL 6) at program start: 0%;
Percent design drawings complete at design review: 59%;
Percent growth in estimated development cost[C]: 14%;
Percent growth in estimated development schedule: 1%.

Program: Global Hawk unmanned aerial vehicle;
Program start: 2/2001;
Formal Milestone I[A] or Milestone A decision review?: No;
Percent technology mature (TRL 6) at program start: 0%;
Percent design drawings complete at design review: 33%;
Percent growth in estimated development cost[C]: 166%;
Percent growth in estimated development schedule: Undetermined.

Program: UH-60M helicopter upgrade;
Program start: 4/2001;
Formal Milestone I[A] or Milestone A decision review?: No;
Percent technology mature (TRL 6) at program start: Not available;
Percent design drawings complete at design review: Not available;
Percent growth in estimated development cost[C]: 151%;
Percent growth in estimated development schedule: 25%.

Program: C-130 Avionics Modernization Program;
Program start: 8/2001;
Formal Milestone I[A] or Milestone A decision review?: No;
Percent technology mature (TRL 6) at program start: 100%;
Percent design drawings complete at design review: Not available;
Percent growth in estimated development cost[C]: 122%;
Percent growth in estimated development schedule: Undetermined.

Program: Joint Strike Fighter;
Program start: 10/2001;
Formal Milestone I[A] or Milestone A decision review?: Yes;
Percent technology mature (TRL 6) at program start: 25%;
Percent design drawings complete at design review: 52%b;
Percent growth in estimated development cost[C]: 30%;
Percent growth in estimated development schedule: 23%.

Program: C-5 Reliability Enhancement and Re-engining Program;
Program start: 11/2001;
Formal Milestone I[A] or Milestone A decision review?: Yes;
Percent technology mature (TRL 6) at program start: 100%;
Percent design drawings complete at design review: 98%;
Percent growth in estimated development cost[C]: 0%;
Percent growth in estimated development schedule: 25%.

Program: Joint Tactical Radio System Cluster 1;
Program start: 6/2002;
Formal Milestone I[A] or Milestone A decision review?: No;
Percent technology mature (TRL 6) at program start: 0%;
Percent design drawings complete at design review: 28%;
Percent growth in estimated development cost[C]: 31%;
Percent growth in estimated development schedule: 44%.

Program: Joint Tactical Radio System Waveform;
Program start: 6/2002;
Formal Milestone I[A] or Milestone A decision review?: No;
Percent technology mature (TRL 6) at program start: Not available;
Percent design drawings complete at design review: Not available;
Percent growth in estimated development cost[C]: 44%;
Percent growth in estimated development schedule: Undetermined.

Program: Advanced Anti-radiation Guided Missile;
Program start: 4/2003;
Formal Milestone I[A] or Milestone A decision review?: No;
Percent technology mature (TRL 6) at program start: Not available;
Percent design drawings complete at design review: Not available;
Percent growth in estimated development cost[C]: 7%;
Percent growth in estimated development schedule: 0%.

Program: Multi-Platform Radar Technology Insertion Program;
Program start: 4/2003;
Formal Milestone I[A] or Milestone A decision review?: No;
Percent technology mature (TRL 6) at program start: 100%;
Percent design drawings complete at design review: 100% b;
Percent growth in estimated development cost[C]: 0%;
Percent growth in estimated development schedule: Undetermined.

Program: Future Combat System;
Program start: 5/2003;
Formal Milestone I[A] or Milestone A decision review?: No;
Percent technology mature (TRL 6) at program start: 19%;
Percent design drawings complete at design review: Not available;
Percent growth in estimated development cost[C]: 48%;
Percent growth in estimated development schedule: 53%.

Program: E-2 Advanced Hawkeye;
Program start: 6/2003;
Formal Milestone I[A] or Milestone A decision review?: No;
Percent technology mature (TRL 6) at program start: 50%;
Percent design drawings complete at design review: 90%;
Percent growth in estimated development cost[C]: 5%;
Percent growth in estimated development schedule: 0%.

Program: Warfighter Information Network-Tactical;
Program start: 7/ 2003;
Formal Milestone I[A] or Milestone A decision review?: No;
Percent technology mature (TRL 6) at program start: 25%;
Percent design drawings complete at design review: Not available;
Percent growth in estimated development cost[C]: 0%;
Percent growth in estimated development schedule: 0%.

Program: Small Diameter Bomb;
Program start: 10/2003;
Formal Milestone I[A] or Milestone A decision review?: Yes;
Percent technology mature (TRL 6) at program start: 100%;
Percent design drawings complete at design review: Not available;
Percent growth in estimated development cost[C]: 0%;
Percent growth in estimated development schedule: 0%.

Program: EA-18G;
Program start: 11/2003;
Formal Milestone I[A] or Milestone A decision review?: No;
Percent technology mature (TRL 6) at program start: 60%;
Percent design drawings complete at design review: 97%;
Percent growth in estimated development cost[C]: 7%;
Percent growth in estimated development schedule: 0%.

Program: Joint Tactical Radio System Cluster 5;
Program start: 4/2004;
Formal Milestone I[A] or Milestone A decision review?: No;
Percent technology mature (TRL 6) at program start: 50%;
Percent design drawings complete at design review: Not available;
Percent growth in estimated development cost[C]: 0%;
Percent growth in estimated development schedule: 2%.

Program: Multi-Mission Maritime Aircraft;
Program start: 5/2004;
Formal Milestone I[A] or Milestone A decision review?: No;
Percent technology mature (TRL 6) at program start: 0%;
Percent design drawings complete at design review: Not available;
Percent growth in estimated development cost[C]: 0%;
Percent growth in estimated development schedule: 0%.

Program: Standard Missile-6 Extended Range Active Missile Block 1;
Program start: 6/2004;
Formal Milestone I[A] or Milestone A decision review?: No;
Percent technology mature (TRL 6) at program start: Not available;
Percent design drawings complete at design review: Not available;
Percent growth in estimated development cost[C]: 0%;
Percent growth in estimated development schedule: 0%.

Program: Aerial Common Sensor;
Program start: 7/2004;
Formal Milestone I[A] or Milestone A decision review?: Yes;
Percent technology mature (TRL 6) at program start: 50%;
Percent design drawings complete at design review: 39% b;
Percent growth in estimated development cost[C]: 45%;
Percent growth in estimated development schedule: 36%.

Program: B-2 Radar Modernization Program;
Program start: 7/2004;
Formal Milestone I[A] or Milestone A decision review?: No;
Percent technology mature (TRL 6) at program start: 100%;
Percent design drawings complete at design review: 84%;
Percent growth in estimated development cost[C]: 0%;
Percent growth in estimated development schedule: 0%.

Program: Patriot/Medium Extended Air Defense System Combined Aggregate 
Program (fire unit);
Program start: 8/2004;
Formal Milestone I[A] or Milestone A decision review?: No;
Percent technology mature (TRL 6) at program start: 83%;
Percent design drawings complete at design review: Not available;
Percent growth in estimated development cost[C]: 0%;
Percent growth in estimated development schedule: 0%.

Program: Mission Planning System;
Program start: 12/2004;
Formal Milestone I[A] or Milestone A decision review?: No;
Percent technology mature (TRL 6) at program start: Not available;
Percent design drawings complete at design review: Not available;
Percent growth in estimated development cost[C]: 0%;
Percent growth in estimated development schedule: 0%.

Sources: DOD (data); GAO (analysis and presentation).

Note: In this table the term "not available" means that GAO had not 
received sufficient data to make an assessment of the given program's 
design and/or technology maturity.

[A] Milestone I was a forerunner to Milestone A, the decision review 
that currently precedes the start of technology development.

[B] Program office projections.

[C] Cost growth is expressed as the percent change in program 
development cost estimates in fiscal year 2005 dollars.

[End of table]

[End of section]

Related GAO Products:

DOD Acquisition Outcomes: A Case for Change. GAO-06-257T. Washington, 
D.C.: November 15, 2005.

Defense Acquisitions: Stronger Management Practices Are Needed to 
Improve DOD's Software-Intensive Weapon Acquisitions. GAO-04-393. 
Washington, D.C.: March 1, 2004.

Best Practices: Setting Requirements Differently Could Reduce Weapon 
Systems' Total Ownership Costs. GAO-03-57. Washington, D.C.: February 
11, 2003:

Best Practices: Capturing Design and Manufacturing Knowledge Early 
Improves Acquisition Outcomes. GAO-02-701. Washington, D.C.: July 15, 
2002.

Defense Acquisitions: DOD Faces Challenges in Implementing Best 
Practices. GAO-02-469T. Washington, D.C.: February 27, 2002.

Best Practices: Better Matching of Needs and Resources Will Lead to 
Better Weapon System Outcomes. GAO-01-288. Washington, D.C.: March 8, 
2001.

Best Practices: A More Constructive Test Approach Is Key to Better 
Weapon System Outcomes. GAO/NSIAD-00-199. Washington, D.C.: July 31, 
2000.

Defense Acquisition: Employing Best Practices Can Shape Better Weapon 
System Decisions. GAO/T-NSIAD-00-137. Washington, D.C.: April 26, 2000.

Best Practices: DOD Training Can Do More to Help Weapon System Programs 
Implement Best Practices. GAO/NSIAD-99-206. Washington, D.C.: August16, 
1999.

Best Practices: Better Management of Technology Development Can Improve 
Weapon System Outcomes. GAO/NSIAD-99-162. Washington, D.C.: July 30, 
1999.

Defense Acquisitions: Best Commercial Practices Can Improve Program 
Outcomes. GAO/T-NSIAD-99-116. Washington, D.C.: March 17, 1999.

Defense Acquisition: Improved Program Outcomes Are Possible. GAO/T- 
NSIAD-98-123. Washington, D.C.: March 17, 1998.

Best Practices: DOD Can Help Suppliers Contribute More to Weapon System 
Programs. GAO/NSIAD-98-87. Washington, D.C.: March 17, 1998.

Best Practices: Successful Application to Weapon Acquisition Requires 
Changes in DOD's Environment. GAO/NSIAD-98-56. Washington, D.C.: 
February 24, 1998.

Best Practices: Commercial Quality Assurance Practices Offer 
Improvements for DOD. GAO/NSIAD-96-162. Washington, D.C.: August 26, 
1996.

FOOTNOTES

[1] In addition to the acquisition policy, the process used by DOD to 
establish program funding, known as the Planning, Programming, 
Budgeting and Execution System (PPBES), and the process used to 
determine system requirements, now called the Joint Capabilities 
Integration and Development System (JCIDS), also impact program 
outcomes. Both processes are currently being studied to determine if 
any changes could be made to improve program performance as it relates 
to funding and requirements.

[2] GAO. Defense Acquisitions: DOD's Revised Policy Emphasizes Best 
Practices, but More Controls Are Needed, GAO-04-53 (Washington, D.C.: 
Nov. 10, 2003).

[3] According to DOD Instruction 5000.2, the concept refinement phase 
is intended to refine the initial concept and develop a technology 
development strategy. Concept refinement ends when the decision 
authority approves a preferred solution resulting from the analysis of 
alternatives and approves the associated technology development 
strategy. After concept refinement, a project enters technology 
development at Milestone A, when the decision maker has approved the 
technology development strategy. The purpose of this phase is to reduce 
technology risk and to determine the appropriate set of technologies to 
be integrated into a full system.

[4] Systems engineering is a technical management tool that provides 
the knowledge necessary to translate requirements into specific, 
achievable capabilities. By using the tools of systems engineering 
during these early phases of concept refinement and technology 
development acquisition decision makers and developers can work 
together to close gaps between requirements and available resources-- 
well before system development starts.

[5] DOD's revised policy emphasizes the importance of reducing 
technology risk and demonstrating technologies in a relevant 
environment (technology readiness level 6) prior to program start. A 
technology readiness level of 6 means the technology should be very 
close to the planned form, fit, and function of its physical 
configuration and that it has been tested or proven to work in a 
relevant environment such as a laboratory. GAO recommends a higher 
level of maturity in its best practice model based on best commercial 
practices. This would require a demonstration of the technology in the 
environment it is expected to be used.

[6] Some programs did not report having a design readiness review but 
did report having a critical design review. Where this was the case, we 
assessed those programs' drawing data at their critical design review.

[7] GAO, TACTICAL AIRCRAFT: Opportunity to Reduce Risks in the Joint 
Strike Fighter Program with Different Acquisition Strategy, GAO-05-271 
(Washington, D.C.: March 15, 2005) and GAO, UNMANNED AERIAL VEHICLES: 
Changes in Global Hawk's Acquisition Strategy Are Needed to Reduce 
Program Risks, GAO-05-6 (Washington, D.C.: November 5, 2004).

[8] GAO, DEFENSE ACQUISITIONS: DOD's Revised Policy Emphasizes Best 
Practices, but More Controls Are Needed, GAO-04-53 (November 10, 2003).

[9] GAO, DEFENSE ACQUISITIONS: Information for Congress on Performance 
of Major Programs Can Be More Complete, Timely, and Accessible, GAO-05-
182 (March 28, 2005).

GAO's Mission:

The Government Accountability Office, the investigative arm of 
Congress, exists to support Congress in meeting its constitutional 
responsibilities and to help improve the performance and accountability 
of the federal government for the American people. GAO examines the use 
of public funds; evaluates federal programs and policies; and provides 
analyses, recommendations, and other assistance to help Congress make 
informed oversight, policy, and funding decisions. GAO's commitment to 
good government is reflected in its core values of accountability, 
integrity, and reliability.

Obtaining Copies of GAO Reports and Testimony:

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains 
abstracts and full-text files of current reports and testimony and an 
expanding archive of older products. The Web site features a search 
engine to help you locate documents using key words and phrases. You 
can print these documents in their entirety, including charts and other 
graphics.

Each day, GAO issues a list of newly released reports, testimony, and 
correspondence. GAO posts this list, known as "Today's Reports," on its 
Web site daily. The list contains links to the full-text document 
files. To have GAO e-mail this list to you every afternoon, go to 
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order 
GAO Products" heading.

Order by Mail or Phone:

The first copy of each printed report is free. Additional copies are $2 
each. A check or money order should be made out to the Superintendent 
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 
more copies mailed to a single address are discounted 25 percent. 
Orders should be sent to:

U.S. Government Accountability Office

441 G Street NW, Room LM

Washington, D.C. 20548:

To order by Phone:

Voice: (202) 512-6000:

TDD: (202) 512-2537:

Fax: (202) 512-6061:

To Report Fraud, Waste, and Abuse in Federal Programs:

Contact:

Web site: www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470:

Public Affairs:

Jeff Nelligan, managing director,

NelliganJ@gao.gov

(202) 512-4800

U.S. Government Accountability Office,

441 G Street NW, Room 7149

Washington, D.C. 20548: