This is the accessible text file for GAO report number GAO-08-619 entitled 'Defense Acquisitions: A Knowledge-Based Funding Approach Could Improve Major Weapon System Program Outcomes' which was released on July 3, 2008. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. Report to the Committee on Armed Services, U.S. Senate: United States Government Accountability Office: GAO: June 2008: Defense Acquisitions: A Knowledge-Based Funding Approach Could Improve Major Weapon System Program Outcomes: Defense Acquisitions: GAO-08-619: GAO Highlights: Highlights of GAO-08-619, a report to the Committee on Armed Services, U.S. Senate. Why GAO Did This Study: The Department of Defense (DOD) expects the cost to develop and procure the major weapon systems in its current portfolio to total $1.6 trillion. With increased competition for funding within DOD and across the federal government, effectively managing these acquisitions is critical. Yet DOD programs too often experience poor outcomes—like increased costs and delayed fielding of needed capabilities to the warfighter. In 2006, this Committee mandated that GAO report on DOD’s processes for identifying needs and allocating resources for its weapon system programs. In 2007, GAO reported that DOD consistently commits to more programs than it can support. This follow-on report assesses DOD’s funding approach, identifies key factors that influence the effectiveness of this approach, and identifies practices that could help improve DOD’s approach. To conduct its work, GAO assessed 20 major weapon programs in DOD’s current portfolio—5 in detail—and reviewed relevant DOD policy and guidance, prior GAO work, and other relevant literature. GAO also reviewed the practices of selected successful companies. What GAO Found: DOD often does not commit full funding to develop its major weapon systems when they are initiated, despite the department’s policy to do so. For a majority of the weapon system programs GAO reviewed, costs have exceeded the funding levels initially planned for and reflected in the Future Years Defense Program (FYDP)—DOD’s investment strategy. To compensate for these shortfalls, DOD makes unplanned and inefficient funding adjustments, like moving money from one program to another, deferring costs into the future, or reducing procurement quantities. Figure: Funding Shortfalls at the Start of Development for Five Major Weapon System Programs: Program: Multi-mission Maritime Aircraft; Level of funding established in the FYDP in the year the program was initiated: 32%; Level of funding the program needed to be fully funded in the initial FYDP: 35%; Funding required beyond the initial FYDP to complete development: 33%; Total: 100%. Program: Warfighter Information Network-Tactical; Level of funding established in the FYDP in the year the program was initiated: 21%; Level of funding the program needed to be fully funded in the initial FYDP: 49%; Funding required beyond the initial FYDP to complete development: 30%; Total: 100%. Program: Future Combat System; Level of funding established in the FYDP in the year the program was initiated: 26%; Level of funding the program needed to be fully funded in the initial FYDP: 41%; Funding required beyond the initial FYDP to complete development: 33%; Total: 100%. Program: Joint Strike Fighter; Level of funding established in the FYDP in the year the program was initiated: 64%; Level of funding the program needed to be fully funded in the initial FYDP: 13%; Funding required beyond the initial FYDP to complete development: 23%; Total: 100%. Program: Global Hawk; Level of funding established in the FYDP in the year the program was initiated: 31%; Level of funding the program needed to be fully funded in the initial FYDP: 48%; Funding required beyond the initial FYDP to complete development: Total: 21%. Source: DOD (data). GAO (analysis and presentation). [End of figure] DOD’s flawed funding process is largely driven by decision makers’ willingness to accept unrealistic cost estimates and DOD’s commitment to more programs than it can support. DOD often underestimates development costs—due in part to a lack of knowledge and optimistic assumptions about requirements and critical technologies. At the same time, DOD’s continued failure to balance its needs with available resources promotes unhealthy competition among programs for funding. This creates incentives for service and program officials to establish requirements that make their particular weapon systems stand out, with less consideration of the resources needed to develop them. Ultimately, DOD tends to push the need for funding to the future rather than limit program length or adjust requirements. The successful commercial companies that GAO has previously reviewed achieve adequate and stable funding for product development programs by following a disciplined, knowledge-based approach to estimating program costs; using manageable development cycles to increase the predictability of funding needs and the likelihood of program success; and using portfolio management practices to make decisions about which programs to pursue. Once programs are approved, these companies firmly commit to fully fund them. What GAO Recommends: GAO is making three recommendations aimed at increasing funding stability and improving acquisition outcomes. DOD believes that current policies and initiatives sufficiently address the first two recommendations, and did not concur with the third. To view the full product, including the scope and methodology, click on [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-08-619]. For more information, contact Michael J. Sullivan at (202) 512-4841 or sullivanm@gao.gov. [End of section] Contents: Letter: Results in Brief: Background: Failure to Commit Full Funding to Weapon Systems Contributes to Poor Acquisition Outcomes: Unrealistic Cost Estimates and a Failure to Balance Needs with Available Resources Underlie DOD's Flawed Funding Approach: Proven Practices Help Ensure Accurate Cost Estimates and Adequate Program Funding: Conclusions: Recommendations for Executive Action: Agency Comments and Evaluation: Appendix I: Objectives, Scope, and Methodology: Appendix II: Comments from the Department of Defense: Related GAO Products: Table: Table 1: Development Cost Estimates and Baselines for 20 Major Weapon System Programs: Figures: Figure 1: Simplified View of PPBE Process: Figure 2: Shortfalls at the Start of Development for Five Major Weapon System Programs: Figure 3: Costs Remaining Versus Annual Appropriations for Major Defense Acquisitions: Figure 4: Notional Comparison of Cost Estimating Uncertainty and Levels of Knowledge at Program Start: Figure 5: Range of Possible Costs Narrows as Knowledge Is Gained: Abbreviations: AEHF: Advanced Extremely High Frequency Satellite: CAIG: Cost Analysis Improvement Group: CARD: Cost Analysis Requirements Description: DAS: Defense Acquisition System: DOD: Department of Defense: EFV: Expeditionary Fighting Vehicle: FCS: Future Combat System: FYDP: Future Years Defense Program: IDA: Institute for Defense Analysis: JCIDS: Joint Capabilities Integration and Development System: JSF: Joint Strike Fighter: JTRS: Joint Tactical Radio System: MDA: Milestone Decision Authority: MMA: Multi-mission Maritime Aircraft: OMB: Office of Management and Budget: OSD: Office of the Secretary of Defense: PPBE: Planning, Programming, Budgeting, and Execution: SAR: Selected Acquisition Reports: SIBRS: Space-Based Infrared System-High: WIN-T: Warfighter Information Network-Tactical: [End of section] United States Government Accountability Office: Washington, DC 20548: July 2, 2008: The Honorable Carl Levin: Chairman: The Honorable John McCain: Ranking Member: Committee on Armed Services: United States Senate: The Department of Defense (DOD) expects the cost to develop and procure the major weapon systems in its current portfolio to total $1.6 trillion, $335 billion of which is expected to be spent over the next 5 years. Effective management of the costs of these acquisitions is critical given the increased competition for funds within the department to support ongoing military operations in Afghanistan and Iraq, as well as growing pressures to reduce overall DOD spending due to the long-term fiscal imbalances facing the federal government. However, many of DOD's major weapon system development programs have experienced poor outcomes--cost increases that add up to hundreds of millions of dollars, schedule delays that add up to years, and capabilities that fall short of what was promised. How DOD manages its weapon system investments has been a matter of congressional concern for many years. In fiscal year 2006, the Senate Armed Services Committee raised concerns about DOD's poor track record with acquisition programs and directed GAO to assess how DOD's processes and practices for identifying requirements and allocating resources affect the department's weapon system acquisition programs. In March 2007, we reported that DOD lacks an effective, integrated portfolio management approach that takes into account all of the department's major weapon system programs and that requires tough decisions commensurate with available resources.[Footnote 1] In short, we noted that DOD commits to more programs than it can support. This report, also done in response to the same Senate mandate, focuses on DOD's funding process and its impact on major acquisitions. Specifically, the report (1) assesses how DOD budgets for and funds its major weapons system acquisition programs, (2) identifies key factors that influence the effectiveness of this approach, and (3) identifies proven processes and practices that could help improve DOD's ability to effectively allocate resources to its acquisition programs. GAO also has ongoing related work specifically assessing DOD's process for identifying and prioritizing warfighting capability requirements. To assess DOD's funding process and to identify key factors that influence the effectiveness of that approach, we reviewed relevant DOD policy guidance, legislation, and academic literature, and assessed cost estimates and budget data for 20 of the 95 major weapons programs in DOD's current portfolio--which represent more than one-third of the total expected cost of DOD's current portfolio of major weapon system programs. To gain further insights into the impact of DOD's funding process on individual programs, we conducted more detailed analysis for five of these programs: Global Hawk, Joint Strike Fighter (JSF), Future Combat System (FCS), Warfighter Information Network-Tactical (WIN-T), and Multi-mission Maritime Aircraft (MMA).[Footnote 2] In addition we interviewed numerous officials from the Office of the Secretary of Defense (OSD) as well as military service cost analysis, budgeting, and acquisition offices. To identify proven cost estimating and budgeting processes and practices that could be used by DOD to improve its resource allocation process, we utilized information from our March 2007 best practices report, and conducted follow-up interviews with officials from three of the five companies that provided input to that report--Eli Lilly, IBM, and Motorola. We also relied on our Cost Assessment Guide, which provides a cost-estimating methodology based on best practices.[Footnote 3] We conducted this performance audit from June 2007 to May 2008 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Results in Brief: DOD often does not commit full funding to its major weapon system acquisition programs, despite the department's policy to "fully fund" programs at the start of system development.[Footnote 4] Of the programs we reviewed, over 75 percent were not fully funded in the Future Years Defense Program (FYDP)--DOD's investment strategy. We found that because programs typically had development cycles that extended beyond the FYDP time frame, the FYDP did not capture their full funding needs. At the same time, program costs exceeded the funding levels initially planned for and reflected in the years covered by the FYDP. To compensate for funding shortfalls, DOD often makes unplanned and inefficient funding adjustments, such as moving money between programs, deferring work and associated costs into the future, or reducing procurement quantities. Ultimately, such reactive practices obscure true program costs and contribute to the instability of many programs and poor acquisition outcomes. DOD's inability to allocate funding effectively to programs is largely driven by the acceptance of unrealistic cost estimates and a failure to balance needs based on available resources. Development costs for major acquisition programs are often underestimated at program initiation--30 to 40 percent in some cases--in large part because the estimates are based on limited knowledge and optimistic assumptions about system requirements and critical technologies. For example, initial development cost estimates for the Army's WIN-T communications system were understated by at least $1.3 billion, or nearly 160 percent, in part because the estimates assumed that commercial-off-the-shelf radio technology would be available. This assumption proved to be wrong. Similarly, JSF's development costs were underestimated by at least $7.1 billion, or around 20 percent, because the initial estimate assumed certain efficiencies that never materialized. These unrealistic cost estimates are developed in an environment where DOD commits to more programs than available resources can support, which promotes unhealthy competition among programs for funding. This competition creates strong incentives for program officials to establish requirements that make their particular weapon systems stand out from others, with less consideration given to the resources that will be needed to develop them. Ultimately, programs tend to push the need for funding to the future rather than limit program length or adjust requirements. Our past work on commercial best practices found that successful companies were able to fully fund their product development programs because they required programs to have realistic, knowledge-based cost estimates; they committed to manageable product increments; and they prioritized programs within resource constraints. To develop realistic cost estimates, these companies followed a disciplined process through which requirements were assessed against available resources at multiple gated reviews prior to committing to a new product. As more knowledge was gained and risk was reduced, costs estimates more accurately reflected true costs. Before initiating product development, for example, Motorola and IBM expected that actual costs would not exceed the latest estimate by more than 5 or 10 percent. To increase the predictability of funding needs--and funding stability--these companies expected development cycles to be manageable (2 to 5 years). Likewise, a study commissioned by DOD in 2006 recommended that programs should be time-constrained with development cycles no longer than 6 years, and DOD is taking steps to pilot this concept. Finally, companies used a portfolio management approach to prioritize investments and allocate resources. Such an approach requires tough decisions about approving or terminating programs. Once programs were approved, these companies made a firm commitment to fully fund them. Ultimately, these practices helped companies avoid committing to more programs than they could afford and allowed them to optimize the return on their investments. To more effectively fund major weapon system acquisition programs and achieve successful outcomes, we are recommending that the Secretary of Defense develop and implement a strategy to bring DOD's current weapon systems portfolio into balance by aligning the number of systems with available resources in the FYDP. In addition, the Secretary should require new programs to have manageable development cycles, realistic cost estimates, and to have planned and programmed full funding for the entire development cycle. Finally, the Secretary should require all cost estimates submitted for funding a program at milestone decisions to be reported as a range of likely costs and reflect the associated levels of risk and uncertainty. DOD partially concurred with our first two recommendations, but believes that they are being sufficiently addressed through current acquisition policies and ongoing initiatives. While we agree that if implemented appropriately, DOD's current policies and initiatives have the potential to contribute to better program outcomes, we have found no evidence of widespread adoption of a knowledge-based process to better ensure adequate funding for programs. DOD agreed with the spirit of our third recommendation--to consider risks at key milestones--however, it did not agree that all cost estimates submitted for funding a program at milestone decisions should be reported as a range of likely costs. In its written comments, DOD stated that a certain method used to calculate cost ranges can produce misleading results. While we do not advocate a specific method for calculating a range of potential costs, we maintain that presenting such a range would provide decision makers with additional knowledge about the level of risk in a proposed program. Background: To plan, execute, and fund its weapon system acquisition programs, DOD relies on three principal decision-making systems: the Joint Capabilities Integration and Development System (JCIDS), which is used to assess gaps in warfighting capabilities and recommend solutions to resolve those gaps; the Defense Acquisition System (DAS), which is used to manage the development and procurement of weapon systems and other equipment; and the Planning, Programming, Budgeting, and Execution (PPBE) process, which is used to allocate resources. While the JCIDS and DAS processes are driven by specific events--such as validating requirements or receiving approval to start development--the PPBE process is calendar driven, taking nearly 2 years to go from planning to the beginning of budget execution. The PPBE process is intended to provide a framework within which DOD can articulate its strategy; identify force size, structure, and needed equipment; set program priorities; allocate resources to individual programs; and assess program performance. Although the different phases of PPBE are considered sequential, because of the amount of time required to develop and review resource requirements, the process is continuous and concurrent with at least two phases ongoing at any given time (see fig. 1). Figure 1: Simplified View of PPBE Process: [See PDF for image] This figure is an illustration of a simplified view of PPBE Process, as follows: 2007 PPBE: Planning: Late 2004 to mid-2005; Programming and budgeting: Mid-2005 to late 2006; Execution (1 to 3 years depending on funding type): Late 2006 to 2008 and beyond. 2008 PPBE: Planning: Late 2005 to mid-2006; Programming and budgeting: Mid-2006 to late 2007; Execution (1 to 3 years depending on funding type): Late 2007 to 2009 and beyond. 2009 PPBE: Planning: Late 2006 to mid-2007; Programming and budgeting: Mid-2007 to late 2008; Execution (1 to 3 years depending on funding type): Late 2008 to 2010 and beyond. 2010 PPBE: Planning: Late 2007 to mid-2008; Programming and budgeting: Mid-2008 to late 2009; Execution (1 to 3 years depending on funding type): Late 2009 to 2011 and beyond. 2011 PPBE: Planning: Late 2008 to mid-2009; Programming and budgeting: Mid-2009 to late 2010; Execution (1 to 3 years depending on funding type): Late 2010 to 2012 and beyond. Source: DOD (data). GAO (analysis and presentation). [End of figure] At the front end of the PPBE process, the Secretary of Defense provides planning guidance to the military services and defense agencies about the capabilities and resources required to deter and defeat threats. The services and agencies, in turn, develop individual "programs" (budgets) based on the planning guidance as well as fiscal guidance provided by the Office of Management and Budget (OMB). The service and agency program budgets are then subjected to a series of leadership reviews in the department--OSD, Director of Program Analysis and Evaluation, Joint Chiefs of Staff, and the Comptroller--and adjustments are made if necessary. Responsibility for managing the PPBE process and ensuring the budget is prepared and submitted to Congress resides with the OSD Comptroller. The PPBE process produces the defense portion of the President's annual budget request to Congress as well as the FYDP--DOD's longer-term investment strategy--which includes the department's resource allocations for the 4 to 5 fiscal years beyond the budget.[Footnote 5] The ultimate objective of the FYDP is to manage funds in a way that provides combatant commanders with the best mix of forces, equipment, and support attainable within fiscal constraints established by OMB. Once complete, the FYDP is expected to provide an aggregate picture of the anticipated force levels and funding needs of individual programs. Data in the FYDP tell the Secretary of Defense, the President, Congress, and the American people what the department expects to invest in and how much it will spend over time. DOD policy requires that the dollars and manpower needed to carry out the first 5-to 6-years of a weapon system acquisition program must be included in the FYDP. The value of the FYDP greatly depends on the accuracy of the cost estimates supporting each individual program. Having a realistic cost estimate provides a basis for accurate budgeting and effective resource allocation, increasing the probability of a program's success in meeting its targets. Two cost estimates are required before a program is approved to start system development (Milestone B) and at the beginning of production (Milestone C)--the service or program office estimate and an independent cost estimate. The independent cost estimate for most major acquisition programs is developed by the Cost Analysis Improvement Group (CAIG). At Milestone B, the Milestone Decision Authority (MDA)[Footnote 6] uses the two estimates to determine the program's official cost baseline and whether the funding reflected in the FYDP is adequate to cover the portion of the estimated costs of the program that fall within the FYDP time period.[Footnote 7] If not, the services are expected to adjust the FYDP to ensure that their programs are fully funded to the approved baseline. To receive Milestone B approval, DOD is required to certify to Congress[Footnote 8] that funding is available to support the portion of the program that falls within the period covered by the FYDP that is submitted during the fiscal year in which the certification is made.[Footnote 9] Our past work has consistently shown that DOD's major weapon system programs do not meet their cost and schedule targets. Since the mid- 1990s, we have studied leading commercial companies in order to identify best practices for developing and producing new products. Taking into account the differences between commercial product development and weapons acquisitions, we articulated a best practices model that relies on increasing knowledge when developing new products, separating technology development from product development, and following an evolutionary or incremental approach to product development.[Footnote 10] This knowledge-based approach requires developers to make investment decisions on the basis of specific, measurable levels of knowledge at critical junctures before investing more money and before advancing to the next phase of acquisition. An evolutionary product development process defines the individual increments on the basis of mature technologies and a feasible design that are matched with firm requirements. The knowledge-based, evolutionary approach in our model is intended to help reduce development risks and to achieve better program outcomes on a more consistent basis. In October 2000, DOD began to significantly revise the department's acquisition policy, adopting a knowledge-based, evolutionary product development approach. DOD's revised policy emphasizes the importance of and provides a good framework for capturing knowledge about critical technologies, product design, and manufacturing processes. In recent years, we have reported that if properly implemented and enforced, DOD's acquisition policy could reduce technical risk at the start of a program and make cost and delivery estimates much more predictable. Failure to Commit Full Funding to Weapon Systems Contributes to Poor Acquisition Outcomes: DOD often does not commit full funding to its major weapon system acquisitions when they are initiated, despite the department's policy to do so. For a majority of the programs we reviewed, costs exceeded the funding levels initially planned for and reflected in the FYDP. To make up for these funding shortfalls, DOD often shifts funds from one program to pay for another, reduces system capabilities, cuts procurement quantities, extends development and procurement schedules, or in rare cases terminates programs. Such actions not only create instability in DOD's weapon system portfolio, they also obscure the true future costs of current commitments, making it difficult to make informed investment decisions. At the beginning of system development, the approved cost estimates for most of the major acquisition programs we reviewed have exceeded the funding initially established in the FYDP. First, our analysis indicates that 14 of the 20 programs had development cycles that extended beyond the FYDP. As a result, the FYDP could not capture all of the estimated funding needed to complete development. Second, the funding established in the FYDP at the beginning of system development for 15 of the 20 programs was less than what they required for that time period.[Footnote 11] For example, at the start of system development in 2003, the initial approved cost estimate for the development portion of the FCS program was about $20 billion, of which $13 billion was needed within the 2003 FYDP time frame--that is for fiscal years 2003 through 2007. However, at that time the 2003 FYDP only contained $5 billion for FCS development. This means that the department had only committed to fund 25 percent of FCS's estimated development cost when the program started, and only 39 percent of the funding needed in its first 5 years. Other programs we reviewed experienced similar shortfalls (see fig. 2). Figure 2: Shortfalls at the Start of Development for Five Major Weapon System Programs: [See PDF for image] Program: Multi-mission Maritime Aircraft; Level of funding established in the FYDP in the year the program was initiated: 32%; Level of funding the program needed to be fully funded in the initial FYDP: 35%; Funding required beyond the initial FYDP to complete development: 33%; Total: 100%. Program: Warfighter Information Network-Tactical; Level of funding established in the FYDP in the year the program was initiated: 21%; Level of funding the program needed to be fully funded in the initial FYDP: 49%; Funding required beyond the initial FYDP to complete development: 30%; Total: 100%. Program: Future Combat System; Level of funding established in the FYDP in the year the program was initiated: 26%; Level of funding the program needed to be fully funded in the initial FYDP: 41%; Funding required beyond the initial FYDP to complete development: 33%; Total: 100%. Program: Joint Strike Fighter; Level of funding established in the FYDP in the year the program was initiated: 64%; Level of funding the program needed to be fully funded in the initial FYDP: 13%; Funding required beyond the initial FYDP to complete development: 23%; Total: 100%. Program: Global Hawk; Level of funding established in the FYDP in the year the program was initiated: 31%; Level of funding the program needed to be fully funded in the initial FYDP: 48%; Funding required beyond the initial FYDP to complete development: Total: 21%. Source: DOD (data). GAO (analysis and presentation). [End of figure] To bring funding commitments more in line with original program baselines, DOD typically makes adjustments in subsequent FYDPs. By that time, however, program cost baselines have often increased--as more knowledge about technologies, design, and manufacturing needs is gained--creating new funding shortfalls. For example, by the third year of the WIN-T program, the program's cost estimate, and therefore funding needs, had increased 72 percent because requirements were unstable and critical technologies were not ready, creating a 26 percent shortfall in the FYDP. Similarly, a 57 percent increase in FCS's cost estimate created an additional 12 percent--or $2 billion-- funding shortfall. In reaction to funding shortfalls in individual programs, DOD often moves funds from other programs,[Footnote 12] reduces capabilities by scaling back requirements, cuts procurement quantities, or extends development schedules. Although rare, programs have also been terminated. Such reactive measures destabilize programs and in many cases increase program costs. For example, as we previously reported, funding for the Air Force's Space-Based Infrared System-High (SIBRS) satellite system was cut in 1998 and 1999 to pay for higher budget priorities, which contributed to a 2-year delay in the program and a breach of the cost baseline. As a result, the Air Force changed SIBRS's procurement strategy, which independent cost estimators calculated would double program costs.[Footnote 13] For the F-22A Raptor program, our past work has noted that the Air Force drastically reduced its planned buys from 648 to 183 as program costs escalated. Similarly, the number of requirements for the Joint Tactical Radio System (JTRS) was reduced or deferred by about one-third as the program encountered development problems. This change had a reverberating effect on several JTRS-dependent efforts--such as the Army's modernization of radios for its helicopters--as those programs had to make adjustments and go forward with alternative, less capable solutions.[Footnote 14] Making these types of adjustments to compensate for inadequate funding obscures future resource requirements and limits the Congress's visibility over DOD's true funding needs. Unrealistic Cost Estimates and a Failure to Balance Needs with Available Resources Underlie DOD's Flawed Funding Approach: For most of the 20 major acquisition programs that we reviewed, initial funding was based on cost estimates that proved to be too low. DOD's funding approach is further compromised by the department's failure to balance requirements with available resources. Because DOD commits to more programs than resources can support, programs often have to compete for funding by overpromising capabilities and by providing low cost estimates as inputs to the funding process. We have previously reported that when such trends go unchecked, Congress is consistently faced with a difficult choice: pull funds from other federal programs to support DOD's acquisitions or accept less warfighting capability than promised.[Footnote 15] Unrealistic Cost Estimates Drive Inaccurate Funding Commitments: The foundation of an accurate funding commitment for a weapon system program should be a realistic cost estimate that is based on a high degree of knowledge about requirements, technology, design, and manufacturing. Realistic estimates also provide a sound basis for setting priorities by allowing decision makers to compare the relative value of one program to another and to make adjustments accordingly. These adjustments could include a decision about whether to proceed with a program, reduce requirements, or defer requirements to a future increment. Most of the 20 programs we reviewed underestimated costs. Unrealistic cost estimates are largely the result of a lack of knowledge, failure to adequately account for risk and uncertainty, and overly optimistic assumptions about the time and resources needed to develop weapon systems. By repeatedly relying on unrealistically low cost estimates, DOD has initiated more programs than its budget can support. Our assessment of cost data for 20 major acquisition programs in DOD's current portfolio found that the majority of these programs were initiated with cost estimates for system development that were too low- -a finding consistent with our prior work and with cost growth patterns reported by RAND, the Institute for Defense Analysis (IDA), and other organizations that conduct defense analyses.[Footnote 16] For 19 of the 20 programs, the independent CAIG estimates were higher than the service estimates--by as much as 139 percent in one case--yet the CAIG estimates for 5 of those programs were still understated by billions of dollars (see table 1). In addition, while 5 of the 20 programs had not reported cost growth as of December 2007, the remaining 15 programs had. For example, the initial CAIG estimate for the Expeditionary Fighting Vehicle (EFV) program was about $1.4 billion compared to a service estimate of about $1.1 billion, but development costs for the EFV system is now expected to be close to $3.6 billion. Similarly, the Army initially estimated that WIN-T development would cost $338 million, but the development program is now expected to cost over $2.0 billion, or $1.7 billion more than initially estimated. Eight of the 20 programs have reported development cost growth of more than 35 percent, resulting in the need for nearly $19 billion in additional funding. Estimates that are this far off the mark do not provide the necessary foundation for sufficient funding commitments. Table 1: Development Cost Estimates and Baselines for 20 Major Weapon System Programs: Program: Global Hawk: Development cost estimate: Service: $905; Development cost estimate: CAIG: $992; Percent difference: 10%; Development cost baselines: Initial: $967; Development cost baselines: Current: $3,515; Percent change: 264%. Program: UH-60M helicopter upgrade; Development cost estimate: Service: $311; Development cost estimate: CAIG: $379; Percent difference: 22%; Development cost baselines: Initial: $311; Development cost baselines: Current: $838; Percent change: 169%. Program: WIN-T; Development cost estimate: Service: $338[A]; Development cost estimate: CAIG: $807; Percent difference: 139%; Development cost baselines: Initial: $796; Development cost baselines: Current: $2,088; Percent change: 162%. Program: C-130 Avionics Modernization; Development cost estimate: Service: $1,020; Development cost estimate: CAIG: $1,175; Percent difference: 15%; Development cost baselines: Initial: $720; Development cost baselines: Current: $1,844; Percent change: 156%. Program: EFV; Development cost estimate: Service: $1,056; Development cost estimate: CAIG: $1,438; Percent difference: 36%; Development cost baselines: Initial: $1,472; Development cost baselines: Current: $3,556; Percent change: 142%. Program: Advanced Extremely High Frequency Satellites; Development cost estimate: Service: $3,031; Development cost estimate: CAIG: $3,175; Percent difference: 5%; Development cost baselines: Initial: $2,923; Development cost baselines: Current: $6,008; Percent change: 105%. Program: Wideband Global SATCOM; Development cost estimate: Service: $296; Development cost estimate: CAIG: $414; Percent difference: 40%; Development cost baselines: Initial: $199; Development cost baselines: Current: $323; Percent change: 62%. Program: Future Combat Systems; Development cost estimate: Service: $20,248; Development cost estimate: CAIG: $27,184; Percent difference: 34%; Development cost baselines: Initial: $20,248; Development cost baselines: Current: $27,955; Percent change: 38%. Program: Joint Strike Fighter[B]; Development cost estimate: Service: $30,500; Development cost estimate: CAIG: $31,476; Percent difference: 3%; Development cost baselines: Initial: $33,939; Development cost baselines: Current: $40,210; Percent change: 18%. Program: COBRA JUDY replacement; Development cost estimate: Service: $1,398; Development cost estimate: CAIG: $1,521; Percent difference: 9%; Development cost baselines: Initial: $1,527; Development cost baselines: Current: $1,626; Percent change: 6%. Program: E-2 Advanced Hawkeye; Development cost estimate: Service: $3,495; Development cost estimate: CAIG: $3,720; Percent difference: 6%; Development cost baselines: Initial: $3,589; Development cost baselines: Current: $3,796; Percent change: 6%. Program: EA-18G; Development cost estimate: Service: $1,707; Development cost estimate: CAIG: $1,795; Percent difference: 5%; Development cost baselines: Initial: 1,797; Development cost baselines: Current: $1,865; Percent change: 4%. Program: VH-71 Presidential Helicopter Replacement Program; Development cost estimate: Service: $3,378; Development cost estimate: CAIG: $3,569; Percent difference: 6%; Development cost baselines: Initial: $3,771; Development cost baselines: Current: $3,859; Percent change: 2%. Program: Joint Land Attack Cruise Missile Defense Elevated Netted Sensor; Development cost estimate: Service: $1,781; Development cost estimate: CAIG: $1,926; Percent difference: 8%; Development cost baselines: Initial: $1,894; Development cost baselines: Current: $1,922; Percent change: 1%. Program: C-5 RERP; Development cost estimate: Service: $1,454; Development cost estimate: CAIG: $1,583; Percent difference: 9%; Development cost baselines: Initial: $1,627; Development cost baselines: Current: $1,630; Percent change: 0[C]. Program: CH-53K Heavy Lift Replacement; Development cost estimate: Service: $3,970; Development cost estimate: CAIG: $4,293; Percent difference: 8%; Development cost baselines: Initial: $4,149; Development cost baselines: Current: $4,095; Percent change: -1%. Program: Longbow Apache III; Development cost estimate: Service: $1,155; Development cost estimate: CAIG: $1,382; Percent difference: 20%; Development cost baselines: Initial: $1,095; Development cost baselines: Current: $1,087; Percent change: -1%. Program: MMA; Development cost estimate: Service: $6,100; Development cost estimate: CAIG: $6,970; Percent difference: 14%; Development cost baselines: Initial: $7,080; Development cost baselines: Current: $6,804; Percent change: -4%. Program: Small Diameter Bomb; Development cost estimate: Service: $416; Development cost estimate: CAIG:$427; Percent difference: 3%; Development cost baselines: Initial: $415; Development cost baselines: Current: $395; Percent change: -5%. Program: Standard Missile 6; Development cost estimate: Service: $1,000; Development cost estimate: CAIG: $992; Percent difference: -1%; Development cost baselines: Initial: $1,009; Development cost baselines: Current: $932; Percent change: -8%. Source: GAO analysis of DOD data. [A] The service's original estimate for the WIN-T program of $338 million was revised prior to Milestone B to align with the CAIG's estimate. [B] JSF data include Air Force and Navy portions of the program only. [C] C-5 RERP costs have increased by almost $4 million. However, due to rounding, the table indicates a 0 percent increase. [End of table] DOD has found similar problems in other acquisition programs. For example, in 2003, a DOD study found that space programs are strongly biased to produce unrealistically low cost estimates throughout the acquisition process.[Footnote 17] The study found that most programs at the time of contract initiation had a predictable cost growth of 50 to 100 percent. The study also found that the unrealistically low projections of program cost and lack of provisions for management reserve seriously distorted management decisions and program content, increased risks to mission success, and virtually guaranteed program delays. Inaccurate cost estimates are often the result of limited knowledge about requirements and technologies. Our best practices work has shown that conducting early disciplined analysis, such as systems engineering,[Footnote 18] builds knowledge that enables a developer to identify and resolve gaps between requirements and available resources before beginning product development. DOD's acquisition policy and guidance emphasize the importance of obtaining knowledge prior to Milestone B--the start of system development--through key analyses and activities--such as conducting an analysis of alternatives (AOA), [Footnote 19] refining requirements, and reducing technology risks. Knowledge gained from these analyses and activities should inform a Cost Analysis Requirements Description (CARD)--a key document that quantifies the program's technical, physical, programmatic, and performance characteristics for developing cost estimates. However, we have frequently reported that DOD programs do not adequately define requirements, mature technologies, or develop an effective acquisition strategy before Milestone B--the point at which cost estimates are approved.[Footnote 20] According to several CAIG analysts, CARDs often lack sufficient detail about planned program content to develop sound cost estimates. One analyst noted that the CARD for the FCS program had to be sent back to the program office because it was too vague. At the same time, programs are expected to deliver the CARD to the CAIG at least 180 days prior to Milestone B, while other supporting documents that contain critical program information--such as the program requirements document--are not required until about 90 days in advance of a milestone review. Many programs conduct analyses of alternatives and other analyses at the same time program cost estimates are being developed. According to a senior CAIG official, such concurrency limits their ability to develop a realistic independent estimate. One senior DOD official also noted that the department's acquisition policy may have encouraged programs to rush to Milestone B, limiting the quality of the data available to develop a program cost estimate. Although DOD's acquisition policy has included an early formal review at Milestone A, this review was not mandatory, and as we have reported in the past, most major acquisition programs have not gone through this early review.[Footnote 21] Cost analysts have also indicated that the lack of comparable products and good historical program data limits knowledge, making it difficult to develop realistic cost estimates. This is true for the majority of DOD's larger and more complex transformational programs that are seeking to provide capabilities that require advanced technologies and equipment that have no historical precedent, like JSF and FCS. In the absence of knowledge, cost estimators must rely heavily on assumptions about system requirements, technology, and design maturity as well as the time and funding needed. However, the level of resources needed for product development is often understated, as we found in several of the programs we reviewed. For example: * FCS: The estimated lines of code needed to support FCS's software development are almost three times original assumptions--from 32 million to 95 million lines of code--leading to an increase in software development costs that now approaches $8 billion. A 2007 IDA report also identified significant additional unquantifiable cost risk due to immature technologies, dependencies on complementary programs, concurrent experimentation and development, and the overall complexity and synchronization of FCS development activities.[Footnote 22] * JSF: JSF assumed that the commonality between the three variants of aircraft and the use of a joint development program, instead of three separate programs, could cut development costs by about 40 percent. However, after development started, significant design issues forced the program to delay development approximately 18 months to conduct unexpected design work. In addition, the assumed commonality between the variants decreased. As we reported in 2005, these two factors contributed to cost increases that nearly eroded all of the assumed cost savings.[Footnote 23] * WIN-T: The Army assumed that the radios and software needed to support the WIN-T system would be commercially available. However, once system development started, the Army learned that the radios and software would require significantly more development and integration than initially anticipated. Further, the Army assumed that WIN-T would be able to meet its portion of the FCS program requirements. However, subsequent changes in the FCS requirements contributed to the need to restructure the WIN-T program. * Global Hawk: The Air Force assumed that validated warfighter requirements could be met with minor additional development to a smaller version, subsequently designated the RQ-4A. However, 1 year after initiating both system development and low-rate initial production, it was determined that a larger airframe and additional, unproven technologies would be needed. As a result, DOD restructured the acquisition strategy to include a second model--designated RQ-4B-- tripling development costs and extending the development cycle from 7 years to 12 years. These examples are consistent with our work over the past several decades, which has raised concerns about the cost and schedule implications of DOD's cost estimating assumptions in its major acquisitions. Our work on DOD's space acquisition programs has found that these programs regularly made overly optimistic cost-estimating assumptions, including assumptions about their ability to define requirements, mature technologies, and secure the funding and other resources needed to develop the system within a specified time frame. [Footnote 24] Similarly, we recently testified that the Navy tends to underestimate the costs needed to construct ships, resulting in unrealistic budgets and large cost increases after ship construction has begun. [Footnote 25] We noted that for two major ship programs, the Navy assumed significant savings based on efficiencies that did not materialize. We linked these optimistic assumptions to cost growth and schedule delays in a number of these programs. Cost estimates that lack knowledge and rely heavily on assumptions have inherently high levels of risk and uncertainty. Conducting quantitative risk and uncertainty analysis provides a way to assess the variability in an estimate. Using this type of analysis, cost estimators can model such effects as a schedule slipping or a key technology failing to materialize, thereby identifying a range of likely costs around an estimate. Presenting decision makers with the range of likely costs around an estimate provides insight into the amount of cost, schedule, and technical risks they are being asked to accept and conveys a level of confidence associated with achieving the proposed estimate. A range of costs also provides a basis for deciding how much funding a program needs to be successful. In developing cost estimates, the services and the CAIG often do not present decision makers with the range of costs around an estimate. [Footnote 26] Instead, they present a single, or point, estimate as the most probable cost, which OSD expects to be a 50 percent chance that actual program costs will be at or below the estimated value. Several recent studies have questioned DOD's approach and recommended establishing estimates with higher levels of confidence under the assumption that the result will be more realistic estimates.[Footnote 27] For example, the Defense Acquisition Performance Assessment Panel recommended adjusting program cost estimates to reflect "high confidence"--defined as an 80 percent chance of completing development at or below the estimated amount. Requiring a higher confidence level could provide a better basis for determining program funding, but only if the quality of the cost estimate is sound and the underlying risk and uncertainty associated with the estimate are accurately captured. If the quality of the estimate is poor to begin with, simply applying a higher confidence level to the estimate will not make it any more realistic. DOD's Failure to Balance Needs with Resources Promotes Unhealthy Competition for Funding: In prior years, we have reported that DOD commits to more programs than its resources can support.[Footnote 28] DOD's failure to balance requirements with available resources promotes unhealthy competition among programs for funding. Ultimately, programs tend to push the need for funding to the future rather than limit program length or adjust requirements. DOD's portfolio of weapon system programs has grown over the past several years at a pace that far exceeds available resources. From 1992 to 2007, the estimated acquisition costs remaining for major weapons programs increased almost 120 percent, while the annual funding provided for these programs only increased 57 percent, creating a fiscal bow wave that may be unsustainable (see fig. 3). If this trend goes unchecked and fiscal pressures to reduce spending continue to grow as expected, Congress will be faced with a difficult choice to either pull funds from other federal programs to support DOD's acquisitions or accept less warfighting capability than promised. Figure 3: Costs Remaining versus Annual Appropriations for Major Defense Acquisitions: [See PDF for image] This figure is a multiple line graph depicting the following information: Fiscal year: 1992; Annual RDTE and Procurement Appropriations: $100; Costs Remaining for Major Defense Acquisitions: $324. Fiscal year: 1993; Annual RDTE and Procurement Appropriations: $91; Costs Remaining for Major Defense Acquisitions: $271. Fiscal year: 1994; Annual RDTE and Procurement Appropriations: $79; Costs Remaining for Major Defense Acquisitions: $253. Fiscal year: 1995; Annual RDTE and Procurement Appropriations: $78; Costs Remaining for Major Defense Acquisitions: $341. Fiscal year: 1996; Annual RDTE and Procurement Appropriations: $78; Costs Remaining for Major Defense Acquisitions: $345. Fiscal year: 1997; Annual RDTE and Procurement Appropriations: $79; Costs Remaining for Major Defense Acquisitions: $331. Fiscal year: 1998; Annual RDTE and Procurement Appropriations: $82; Costs Remaining for Major Defense Acquisitions: $280. Fiscal year: 1999; Annual RDTE and Procurement Appropriations: $89; Costs Remaining for Major Defense Acquisitions: $233. Fiscal year: 2000; Annual RDTE and Procurement Appropriations: $94; Costs Remaining for Major Defense Acquisitions: $309. Fiscal year: 2001; Annual RDTE and Procurement Appropriations: $104; Costs Remaining for Major Defense Acquisitions: $234. Fiscal year: 2002; Annual RDTE and Procurement Appropriations: $111; Costs Remaining for Major Defense Acquisitions: $586. Fiscal year: 2003; Annual RDTE and Procurement Appropriations: $137; Costs Remaining for Major Defense Acquisitions: $579. Fiscal year: 2004; Annual RDTE and Procurement Appropriations: $148; Costs Remaining for Major Defense Acquisitions: $722. Fiscal year: 2005; Annual RDTE and Procurement Appropriations: $165; Costs Remaining for Major Defense Acquisitions: $745. Fiscal year: 2006; Annual RDTE and Procurement Appropriations: %147; Costs Remaining for Major Defense Acquisitions: $841. Fiscal year: 2007; Annual RDTE and Procurement Appropriations: $160. Costs Remaining for Major Defense Acquisitions: $767. Source: DOD (data). GAO (analysis and presentation). [End of figure] DOD's ability to prioritize needs within available resources is hampered because its processes and management structures for determining capability needs and allocating resources are not effectively integrated. Capability needs are formally identified and validated through JCIDS, which does not account for the resources that will be needed to meet those needs. Instead, resources are assessed and allocated through the PPBE process, which as we have reported in the past, is service-centric and does not effectively link resources to capabilities. In addition, the PPBE and JCIDS processes are led by different organizations within DOD, as is the process for executing acquisitions, making it difficult to hold any one person or organization accountable for saying no to a proposed program or for program outcomes. DOD's failure to balance its needs with available resources promotes unhealthy competition among and within the services to get funding for new programs and to sustain funding for existing ones. This competition for funding creates strong incentives for service and program officials to establish requirements that make their particular weapon systems stand out from others, with less consideration given to the resources that will be needed to develop them. Because DOD's overall funding must fit within fiscal constraints of the FYDP that are established by OMB, services place a high priority on the appearance of affordability. To maintain a competitive edge, services develop cost estimates that will fit within established funding levels. Also, because of the length of the PPBE process, initial funding for a program is often established in the FYDP before the program's initial cost estimated is developed and approved. In addition, program officials are motivated to initiate a weapon system development program because once initiated a program is in a more competitive position to attract high levels of funding and management support. Many of the weapon system program managers who responded to a GAO survey in 2005 cited this competitive funding environment as a key obstacle to their ability to effectively manage their programs.[Footnote 29] For example, program managers provided comments such as the following on competition for funding: * "OSD has reduced funding without any understanding or appreciation for program impacts. Funding cuts appear arbitrary." * "OSD's near-term execution year focus results in great instability. In reality, it should provide more strategic vectors for the department instead of short-term adjustments to fix more tactical-level funding needs." * "The service and OSD typically cut programs to pay top down bills." * "There is no such thing as funding stability in DOD. Funding reductions and program stretch-outs are the norm due to top down fiscal bills that occur during the execution year." * "Unstable funding results in pressure to do aggressive things in order to minimize the impact of budget cuts on schedule and performance. I believe this has been a major factor in recent … program execution problems." * "When funding gets tight, we have been considered a bill payer for others, even if it has "broken" our program." OSD reviews program budgets and makes adjustments toward the end of the PPBE cycle--often only 2 or 3 months before the budget is submitted to Congress. For example, in December 2004, OSD cut $30 billion from the 2006 FYDP for many of its major acquisition programs, including prominent aircraft programs like the F-22A Raptor, the V-22 Osprey, and the C-130J Hercules. Attempting to balance investments this late in the process often leads to additional churn in programs and encumbers efforts to meet strategic objectives and joint needs. Instead of rethinking performance requirements and making needed trade- offs within and among programs, DOD and the services often pursue ambitious solutions and technologies that result in cost growth and the need for unplanned funding. Ultimately, DOD often defers costs into the future--beyond the FDYP--expecting that additional funding will become available when it is needed. According to DOD and service officials, what often ends up happening is that programs have to extend their schedules to spread costs out over time or reduce requirements to hold costs down. These strategies ultimately result in delaying the delivery of programs or providing the warfighter with less capability than promised. Proven Practices Help Ensure Accurate Cost Estimates and Adequate Program Funding: Successful commercial companies that we have previously reviewed achieved adequate and stable funding for product development programs by following practices that we have recognized as best practices for estimating and managing program costs.[Footnote 30] In contrast to DOD, these companies followed a disciplined, knowledge-based approach to estimating program costs. Prior to initiating product development, these companies expected cost estimates to be refined based on increasing product knowledge and assessed the estimates at multiple gated reviews. At the same time, the companies expected proposed programs to have a manageable development cycle, which increases the predictability of funding needs and the likelihood of program success. Understanding that resources are limited, the companies used portfolio management practices to make tough decisions about which programs to pursue. Once programs were approved, these companies firmly committed to fully fund them--which they often achieved by committing resources incrementally to successive phases of a program. By making these incremental commitments, these companies were able to limit the disruptions that can be caused by poorly performing programs, and thus maintained a high degree of stability within and among other programs. A Disciplined, Knowledge-Based Approach and Manageable Development Cycles Are Key to Realistic Cost Estimates: As part of our best practices work on successful product development, we have found that following a disciplined, knowledge-based approach that reduces risk and uncertainty over time is integral to developing realistic cost estimates. The successful companies we have reviewed conducted multiple management reviews prior to initiating product development to assess the business case of each proposed product, including its cost estimate. They typically expected cost estimates to be developed early on and refined over time as knowledge is gained. In addition, they expected estimates to be transparent because it allowed them to make informed decisions about the risks and uncertainty associated with proposed programs and whether they should pursue or cancel them. While these companies acknowledged that early in a program's life cycle less is known about the product's design and technologies, they still expected to receive rough cost estimates at that stage. They expected the estimates to be revised and more precise as knowledge was gained. Before committing to product development, they expected the level of uncertainty in the estimates to be low. Our Cost Assessment Guide similarly emphasizes the need to refine cost estimates based on knowledge gained over time and account for risk and uncertainty in the estimates. Early cost estimates are more uncertain because less is known about requirements and the opportunity for change is greater. As more knowledge is gained, programs can retire some risk and reduce the potential for unexpected cost and schedule growth. The best practice is not to commit to product development until the level of knowledge is high and the level of uncertainty is low. However, our past work has found that DOD typically commits to starting programs with low levels of knowledge and high levels of uncertainty (see fig. 4). Figure 4: Notional Comparison of Cost Estimating Uncertainty and Levels of Knowledge at Program Start: [See PDF for image] This figure is a matrix depicting a notional comparison of cost estimating uncertainty and levels of knowledge at program start. Knowledge moves from low to high on the horizontal axis, and cost estimating uncertainty moves from low to high on the vertical axis. Noted on the matrix: Typical DOD program start: Occurs with high cost estimating uncertainty and low knowledge. Best practice program start: Occurs with high knowledge and low cost estimating uncertainty. Source: GAO. [End of figure] Despite having high levels of uncertainty, DOD commits to development programs based on point estimates that are expected to represent most likely costs. The Air Force's Cost Risk and Uncertainty Analysis Handbook notes that decision makers need point estimates when preparing and managing a budget because programs are funded and executed using discrete dollars, not ranges of dollars. However, DOD's estimates often prove to be understated by as much as 30 to 40 percent. To make more informed investment decisions, cost estimating best practices call for estimating a range of possible costs around a point estimate to provide information about the levels of uncertainty and confidence. As proposed programs gain more knowledge and progress through the phases leading up to the start of product development, these ranges should narrow (see fig. 5). According to cost-estimating experts and representatives of one of the commercial companies we spoke with, it is better to overestimate than underestimate costs. Figure 5: Range of Possible Costs Narrows as Knowledge Is Gained: [See PDF for image] This figure s an illustration of how the range of possible costs narrows as knowledge is gained. Prior to the concept refinement phase, the margin of overestimating is extremely high, and the margin for underestimating is at its' highest. During the concept refinement state and the concept definition phase, these margins narrow significantly. During the product development phase, bot margins continue to narrow and approach cost estimating certainty. Source: GAO. [End of figure] Officials in Motorola's Government and Enterprise Mobility Solutions business unit have established a rigorous review process, within which proposed products are expected to provide cost estimates that fall within an established range at successive review gates. At the first review gate, the range is generous, allowing for estimates to be as much as 75 percent too high and 25 percent too low. As the proposed program moves through review gates and more becomes known about requirements, technologies, and design, the established range narrows. At product development initiation, the cost estimate is expected to be no more than 10 percent higher and 5 percent lower than what actual costs will be. IBM similarly allows products to deviate from their original estimates as long as the deviation is within agreed-upon limits, which are established in a contract between senior management and project managers.[Footnote 31] Product development teams are expected to execute according to the contract. According to IBM officials, program cost growth of more than 5 or 10 percent is generally not acceptable. Successful companies have found that a relatively short, manageable development cycle is a hallmark of an executable program. Officials at Motorola told us that cycle time is one of the key metrics they use when making decisions about what programs to pursue. They noted that development programs longer than 2 or 3 years are not likely to be initiated because the increased uncertainty would make it too difficult to accurately plan for and execute the programs. This is consistent with what we found at other successful commercial companies as well. By constraining development cycles, it is easier to more accurately estimate costs, and with more accurate cost estimates companies are able to more precisely predict the future funding needs and effectively allocate resources. In 1998, the Under Secretary of Defense for Acquisition, Technology and Logistics stated that the department's objective must and will be to achieve acquisition cycle times no longer than 5 to 7 years, noting that long acquisition cycle times for major defense programs lead to higher costs and diminished military effectiveness. DOD's acquisition policy, revised in 2003, suggests that system development should be limited to a manageable time frame--about 5 years. An assessment of DOD's acquisition system commissioned by the Deputy Secretary of Defense in 2006 similarly recommended that programs should be time- constrained with development cycles no longer than 6 years from Milestone A to low-rate initial production.[Footnote 32] According to the assessment, a time-constrained development cycle can reduce pressure on investment accounts and increase funding stability for all programs. With development cycles of 6 years or less, programs could be fully funded within the FYDP time frame. In addition, constrained cycle times would force programs to conduct more detailed systems engineering analyses, increasing the likelihood that their requirements can be met with available resources. We recently reported that while there are isolated examples of DOD programs with cycle times shorter than 5 years, the majority of programs were initiated with much longer cycle times. In some cases, these longer cycle times have been extended. [Footnote 33] For example, FCS was expected to be a 7.5-year development effort but is now a 12-year program and will likely be extended again. Unconstrained and lengthy cycle times promote program funding instability--especially when considering DOD's tendency to change requirements and funding as well as frequent changes in leadership. A Portfolio-Based Investment Approach with Incremental Commitments Supports Adequate and Stable Program Funding: To ensure that resources are available to fully support their programs, successful commercial companies we reviewed used an integrated portfolio management approach to make incremental investment decisions. In making these decisions, the companies looked across their entire product portfolio within the context of available resources to ensure that they were pursuing a balanced mix of products that can optimize the return on their investment. Once a decision was made to pursue a development program, the companies committed to fully funding the first phase of that program, and made similar commitments prior to initiating each successive phase. Before programs were approved to enter the next phase of product development, Eli Lilly and IBM required program officials and management to sign a contract that represented the program's promise to deliver a given product that would meet the customers' needs within established cost and time constraints. These contracts also represented the ongoing commitment of management to provide the necessary funding and other resources needed to ensure that the program could be successfully executed. By making incremental commitments companies are better positioned to effectively manage and maintain stability in programs. For companies like Eli Lilly--which tends to have development cycles of 10 to 15 years on average--the practice of making short-term, incremental commitments mitigates many of the cost, schedule, and performance risks inherent in long development programs. When Eli Lilly decides to pursue a new drug development program, management and the project team enter into a contract that identifies deliverables, time frames, and the costs to get to the next milestone. The contract represents management's commitment to fund the entire phase. IBM similarly allocates funding in increments. For example, at the initial review gate, funding is allocated to the product development team to support the development of a sound business plan, which is expected to be presented at the next review gate before any further funding commitment is made. Making incremental commitments also allows companies the flexibility to effectively manage their portfolio. As they assess the value and progress of each investment at multiple points throughout product development, companies are able to make tough decisions to defer or terminate programs and rebalance their portfolio. Terminating low-value or poor-performing programs is important to the successful companies we spoke with, because each dollar spent on a failing program is one less dollar available for use elsewhere in the company. As a result, they emphasize the need to make these tough decisions early. For example, at Eli Lilly projects are terminated at early points in the review process when it is determined that their critical success factors cannot be achieved. Because Eli Lilly's projects typically have a high degree of technical risk, only about 1 percent of those that start early development actually make it to the marketplace. We have found that successful portfolio management requires a strong governance structure with committed leadership that empowers portfolio managers to make decisions about the best way to invest resources and holds those managers accountable for the outcomes they achieve. Because portfolio managers are on the front line, several of the companies we have reviewed empower these managers to make product investment decisions and then hold them accountable for outcomes, not just for individual products but also for the overall performance of the portfolio. These organizations underscore the importance of holding individuals accountable, aligning performance expectations with organizational goals, and cascading those expectations down to lower levels. In contrast, DOD approves proposed programs without adequately considering its overall portfolio and commits to programs with less knowledge of cost and feasibility. Moreover, DOD lacks the accountability that commercial companies emphasize is necessary to ensure successful outcomes. Consequently, DOD starts more programs than current and likely future resources can support and has less assurance that its investment decisions address the right mix of warfighting needs. Conclusions: At a time when the federal budget is strained by spending needs for a growing number of national priorities, it is imperative that DOD get the best value for every dollar of its significant investments. Yet DOD has more major weapons system programs in its portfolio than it can afford. All too often, these programs incur cost increases well beyond original funding levels and significant delays in delivering weapon systems to the warfighter. These outcomes are the direct result of DOD's failure to prioritize its needs within resource constraints and a funding process that allows programs to go forward with unpredictable cost estimates and lengthy development cycles--not a sound basis for allocating resources and ensuring program stability. Successful commercial companies we have reviewed recognize the importance of having adequate knowledge about requirements and available resources before initiating a development program. These companies have learned that disciplined processes with early management review points, knowledge-based cost estimates, manageable product development cycles, and a portfolio approach to funding product development efforts are key to achieving program success. While DOD has recognized that more accurate cost estimates coupled with manageable development cycles could improve program outcomes, it has yet to take action. Until DOD revises its policies and processes to address funding imbalances and reform its funding approach, programs will continue to experience instability and delayed delivery of needed capabilities to the warfighter. Recommendations for Executive Action: To better ensure adequate funding for DOD's major weapon system acquisition programs and to increase the likelihood of achieving successful outcomes, we recommend that the Secretary of Defense take the following three actions: * Develop and implement a strategy to bring the department's current portfolio into balance by aligning the number of programs and the cost and schedule of those programs with available resources. In developing and implementing a strategy, the department should determine ways to prioritize needs and identify whether the budget and the FYDP should be increased to more accurately reflect the actual costs of current programs or whether the portfolio of current programs should be reduced and lower-priority programs terminated to match available resources. * Require that all new programs have manageable development cycles, realistic cost estimates, and have planned and programmed full funding for the entire development cycle. * Require all cost estimates submitted for funding a program at milestone decisions to be reported as a range of likely costs and reflect the associated levels of risk and uncertainty. At Milestone A, require estimates that allow for a wide range of likely costs. At Milestone B, require estimates that, based on knowledge gained, are more precise--in line with best practice standards. Agency Comments and Our Evaluation: In written comments on a draft of this report, DOD partially concurred with our first and second recommendations and non-concurred with the third. DOD's partial concurrences are rooted in the belief that its current policies and initiatives address our recommendations. We agree that aspects of DOD's current policies appear consistent with our recommendations, and the initiatives could contribute to better program outcomes if implemented appropriately. However, we have found no evidence of widespread adoption of these policies or any other process that would better ensure adequate funding for DOD's major weapon system acquisition programs and increase the likelihood of achieving successful outcomes. DOD partially concurred with our first recommendation--that the Secretary of Defense develop and implement a strategy to bring the department's current portfolio into balance by aligning the number of programs and the cost and schedule of those programs with available resources. In its written response, DOD identified the 2003 restructuring of the PPBE process as one initiative it has taken to address the mismatch between program commitments and available resources. Yet, since the restructuring, more than 5 years--and several budget cycles--have lapsed, and the department is still committed to more programs than it can support. While DOD notes that seeing the effects of process changes will take years, the department does not indicate how many additional years are needed to see positive results or what incremental changes it should be held accountable for. In the meantime, the department continues to risk tax dollars and delaying the delivery of needed capabilities to the warfighter. The department also cites several other, more recent initiatives like Capability Portfolio Management, Capital Accounts, and Configuration Steering Boards that are intended to balance needs with resources; drive more realistic, cost-effective plans and budgets; limit requirements growth; and improve oversight. While we believe that these initiatives, like many before them, are well intentioned, DOD has not established indicators to measure their success, and it is unclear how the initiatives will help bring DOD's current portfolio into balance. Further, we are concerned that they do not go far enough to address the systemic cultural and structural problems identified in this report. DOD states that "external influences" can cause turbulence in program execution. Specifically, the department notes that it does not control the amount of Total Obligation Authority (TOA) available in the FYDP or how much funding is appropriated by Congress. While we acknowledge that decisions made by OMB and Congress can directly affect program funding, we believe that our recommendation is sound regardless of the total amount of funding provided to the department. As we have recently reported, substantially more funding has been committed to develop new weapon systems since 2000, yet cost overruns and schedule delays have increased even more. By focusing on external influences DOD misses the main point of our recommendation--to align the number of programs in the current portfolio and the cost and schedule of those programs with available resources. DOD also partially concurred with our second recommendation--to require all new programs to have manageable development cycles, realistic cost estimates, and full funding for the entire development cycle. Again, the department's partial concurrence is based on its belief that certain DOD policies and initiatives respond to this recommendation. We agree that some DOD policies and initiatives emphasize a knowledge- based approach to acquiring weapon systems, but acquisition officials do not effectively implement these policies. DOD programs are often initiated with development cycle times that are much longer than the 5 years the department's policy suggests. As we note in this report, programs with development cycles of 6 years or less could be fully funded in a single FYDP and would be less likely to experience funding instability. DOD further commented that the current acquisition policy requires cost estimates and, where required, independent cost estimates to be completed at key program milestones and full funding to be programmed prior to Milestone B. However, we found that because DOD's cost estimates at key milestones are based on limited knowledge and optimistic assumptions about requirements and technology, they often significantly underestimate true costs and do not provide the necessary foundation for making accurate funding commitments. In addition, our analysis of DOD's selected acquisition reports and budget data found that DOD often does not commit full funding to its major weapon system acquisitions when they are initiated, despite the department's policy to do so. While the department generally agreed that product development cycles should be "manageable," it questioned the comparability of DOD to the private sector firms we reviewed with regard to industry goals, incentives, products, and services. This misses the point. We are recommending that development cycles for all new DOD programs be manageable--a guiding principle that the successful commercial companies we have reviewed follow to enable better program outcomes. The department's own acquisition policy suggests development cycles of about 5 years--an objective the department established a decade ago. Similarly, a recent study commissioned by DOD recommended that programs should be time-constrained with development cycles no longer than 6 years. In general, DOD indicated an interest in obtaining more insight into our methodology. Much of this information, including information about the companies and commodities that we have reviewed in the past, can be found in most of the works cited in the related GAO products section of this report. DOD did not concur with our third recommendation. While DOD agreed that risks should be considered at key milestones, it did not agree that cost estimates submitted at milestone decisions should be reported as a range of likely costs--a wide range at Milestone A based on limited knowledge, narrowing to a more precise range at Milestone B as more knowledge is gained. The CAIG accounts for cost risk by examining program schedule durations, technical risks, contract vehicles, incentives, and management structures. However, DOD's current practice of presenting a single point estimate as the "most likely cost" provides decision makers with limited insight into these risks. DOD also cited concerns that a certain method for calculating cost ranges can produce misleading results. While we do not advocate a specific method for calculating a range of potential costs, we maintain that presenting such a range would provide decision makers with additional knowledge about the level of risk in a proposed program. Finally, DOD states that our report does not consider certain causal factors bearing on the estimates we cite in table 1. It is unclear how DOD came to this conclusion. Our report states that inaccurate cost estimates are often the result of limited knowledge about requirements and technologies. We also note that conducting early disciplined analyses, such as systems engineering, builds knowledge that enables a developer to identify and resolve gaps between requirements and available resources before beginning product development. As knowledge increases, uncertainty and associated risks in the cost estimate decrease. Therefore, we believe that following a disciplined knowledge- based process--a recommendation we have made repeatedly in our reviews of DOD's major weapon system programs--would obviate most of the causal factors DOD cites, including "incomplete, error-full, or volatile requirements or program specification" and "engineering change proposals." DOD's written comments are reprinted in appendix II. The Department also provided technical comments, which were incorporated as appropriate. We are sending copies of this report to the Secretary of Defense; the Secretaries of the Air Force, Army, and Navy; and the Director of the Office of Management and Budget. We will provide copies to others on request. This report will also be available at no charge on GAO's Web site at [hyperlink, http://www.gao.gov]. If you have any questions about this report or need additional information, please contact me at (202) 512-4841 or sullivanm@gao.gov. Key contributors to this report were John Oppenheim, Assistant Director; Travis Masters; Keith Hudson; Victoria Klepacz; Karen Sloan; Karen Richey; and John Krump. Signed by: Michael J. Sullivan: Director, Acquisition and Sourcing Management: [End of section] Appendix I: Objectives, Scope, and Methodology: This report assesses the extent to which the Department of Defense's (DOD) resource allocation approach supports stability within and across its major weapon system acquisition programs. Specifically, our objectives were to (1) assess how DOD budgets for and funds its major weapon system acquisition programs, (2) identify key factors that influence the effectiveness of this approach, and (3) identify proven processes and practices that could help improve DOD's ability to effectively allocate resources to its acquisition programs. To assess the effectiveness of DOD process for funding for its major weapon system acquisition programs, we compared initial approved cost estimates for 20 of the 95 major weapons programs in DOD's current portfolio to the amount of development funding contained in each program's budget documentation from the year in which development started--which covers the initial Future Years Defense Program (FYDP) time frame. The 20 programs we reviewed initiated system development from 1999 through 2006 and represent almost 35 percent of the total expected cost of DOD's current portfolio of major weapon system programs. Due to data limitations, we were not able to assess a number of programs, including several major ship acquisitions, in our cost estimate and funding analysis. While our findings may not be generalized to all of DOD's acquisition programs, we believe that capturing more than one-third of DOD's planned investment is significant and provides insights into DOD's resource allocation approach for its major weapon system programs. To gain further insights into the impact of DOD's funding process on individual programs, we conducted more detailed analysis for 5 high-profile programs, which represent the nearly 75 percent of DOD's planned investment in the 20 programs: Global Hawk, Joint Strike Fighter (JSF), Future Combat System (FCS), Warfighter Information Network-Tactical (WIN-T), and Multi- mission Maritime Aircraft (MMA).[Footnote 34] We obtained cost and baseline data from DOD's selected acquisition reports (SAR) submitted through December 2007, FYDP and program funding data from the President's Budget exhibits through fiscal year 2008, and program cost estimates from the Office of the Secretary of Defense's (OSD) Cost Analysis Improvement Group (CAIG) and the service cost analysis centers developed to support each program's Milestone B review. Our analysis began with cost and funding data from the fiscal year that each program entered system development--received Milestone B approval--and ended with the data reported in the December 2007 SARs. We also examined and compared individual program cost estimates from the CAIG, service cost analysis centers, and program offices to determine how they differed from current program costs. To better understand how DOD funds its weapon system acquisitions and to identify the key factors that influence its effectiveness, we reviewed DOD Acquisition Regulations, DOD Financial Management Regulations, Title 10 of the United States Code, Planning, Programming, Budgeting, and Execution guidance from Management Information Directive- 913, as well as individual military service budgeting policies and guidance. We also met with knowledgeable officials from the OSD CAIG; OSD Program Analysis and Evaluation; the Office of the Under Secretary of Defense for Acquisition, Technology and Logistics; the military service cost analysis centers; military service planning and programming organizations; the Defense Acquisition University; the Brookings Institution; the Institute for Defense Analysis; and the Naval Postgraduate School. To identify proven cost estimating and budgeting processes and practices that could be used by DOD to improve its resource allocation process, we relied on our prior work in best practices at successful commercial companies and reviewed documentation and interviews from previous GAO best practices work. We also conducted follow-up interviews with key officials from Eli Lilly, IBM, and Motorola [Footnote 35] to gain a better understanding of how they prioritize projects for funding and ensure that each product receives and maintains adequate funding. We also relied on our Cost Assessment Guide--which provides a cost-estimating methodology based on best practices--and numerous other GAO products, including our extensive body of best practices work. We conducted this performance audit from June 2007 to May 2008 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. [End of section] Appendix II: Comments from the Department of Defense: Office Of The Under Secretary Of Defense: Acquisition, Technology And Logistics: 3000 Defense Pentagon Washington, DC 20301-3000 June 13, 2008: Mr. Michael J. Sullivan: Director, Acquisition and Sourcing Management: U.S. Government Accountability Office: 441 G Street, N.W. Washington, DC 20548: Dear Mr. Sullivan: This is the Department of Defense (DoD) response to the GAO draft report GAO-08-619, "Defense Acquisitions: A Knowledge-Based Funding Approach Could Improve Major Weapon System Program Outcomes," dated May 15, 2008 (GAO Code 120665). The Department partially concurs with two recommendations and nonconcurs with the other draft report recommendation. The rationale for DoD's position is included in the enclosure. The Department appreciates the opportunity to comment on the draft report. Technical comments were provided separately. My point of contact for this effort is Mr. Joseph A. Alfano, 703-697-3343. Sincerely, Signed by: Dr. Nancy L. Spruill: Director: Acquisition Resources & Analysis: Enclosure: As stated: GAO Draft Report Dated May 15, 2008: GAO-08-619 (GAO Code 120665): "Defense Acquisitions: A Knowledge-Based Funding Approach Could Improve Major Weapon System Program Outcomes" Department Of Defense Comments To The GAO Recommendations: Recommendation 1: The GAO recommends that the Secretary of Defense develop and implement a strategy to bring the department's current portfolio into balance by aligning the number of programs and the cost and schedule of those programs with available resources. In developing and implementing a strategy, the Department should determine ways to prioritize needs and identify whether the budget and the Future Years Defense Program should be increased to more accurately reflect the actual costs of current programs or whether the portfolio of current programs should be reduced and lower priority programs terminated to match available resources. (p. 28/GAO Draft Report) DOD Response: Partially concur. The Department goes to great lengths in order to develop overall investment strategy and guidance, then flows that guidance to the appropriate levels, and then prioritizes requirements in order to "align the number of programs and the cost and schedule of those programs with available resources." The restructure of the Planning, Programming, and Budgeting System into the Planning, Programming, Budgeting, and Execution process was one of several initiatives the Department has taken towards this end; however, positive results from these types of process changes will take years and experience across several programs in order to validate their positive effects. Yet while the Department has control over its programming and budgeting processes (within limits set by the Office of Management and Budget) it does not control the amount of Total Obligation Authority available over the Future Years Defense Program, nor how much funding is appropriated by Congress. At times, these external influences will also cause turbulence in program execution. Over the last several years the Department has taken on several initiatives to address this GAO recommendation. A recent Defense initiative known as Capability Portfolio Management is intended to provide an enterprise-level, horizontal (cross-component) view of the Department to better balance and harmonize the Departments capability needs with existing and planned force management and development efforts and produce strategically aligned outcomes optimized for the enterprise. As of February 2008, Department leadership has established nine Capability Portfolio Managers aligned to the nine Tier 1 Joint Capability Areas (the Department's capabilities language), formalized four and is testing the other five. Capability Portfolio Managers are focused on highlighting opportunities to better integrate, coordinate and synchronize programs to strategic intent and capability priorities within resource and time constraints. With respect to budgeting, more realistic, cost-effective plans and budgets are goals for the integrated Planning, Programming, Budgeting and Execution process. Establishing Capital Accounts will result in more stable, predictable acquisition lifecycle management program execution, and the process would benefit from this guaranteed funding stream. Pilot programs provide valuable lessons learned to apply capital accounts to a wide variety of acquisition and sustainment programs. Current initiatives include coordination with the Office of Management and Budget Program Assessment Rating Tool and establishment of an authoritative financial information source by integrating transactional-level accounting data. Further, Configuration Steering Boards have been implemented across all major programs to limit cost and requirements growth through senior level evaluation of trade-offs and opportunities to descope programs and seek savings. In the oversight of Major Defense Acquisition Programs and the Defense Acquisition Board process, multiple initiatives are being implemented to improve and standardize the milestone decision process. The Defense Acquisition Executive Summaries system is being reengineered to ensure visibility for senior leaders. The system will help facilitate information sharing, accurate input of data, and tracking of negative trends to focus on problem areas. Systems to model time and new technology factors are being developed to budget effectively to changing needs in a rapidly evolving technological development environment. Recommendation 2: The GAO recommends that the Secretary of Defense require that all new programs have manageable development cycles, realistic cost estimates, and have planned and programmed full funding for the entire development cycle. (p. 28/GAO Draft Report) DOD Response: Partially concur. Current DoD acquisition policy, as articulated in DoDI 5000.2, establishes an evolutionary, event-based acquisition approach as the Department's preferred strategy for rapid acquisition of mature technology for the user. Additionally, current policy requires cost estimates and, where required, independent cost estimates to be completed at key program milestones and full funding to be programmed prior to Milestone B. These existing policies are consistent with the GAO's recommendation. Additionally, the Department has recently introduced a number of new policies that we believe will substantively contribute to more predictable program outcomes: * Our prototyping and competition policy requires program managers to plan for two or more competing teams to produce prototypes of key system elements up to or through Milestone B. The intent is to reduce technical risk, validate cost estimates, evaluate manufacturing processes, and refine requirements. * We have established Configuration Steering Boards for all ACAT I programs to review proposed requirements changes, a significant contributor to increased cost and extended development cycles, and any technical configuration changes that have the potential to result in increases to cost and or lengthier schedules. Such changes will generally be rejected, deferring them to future development increments. Finally, the Department is considering policy initiatives that are designed to reduce program instability and improve schedule and cost predictability. These initiatives include: (1) moving the Preliminary Design Review before Milestone B, to improve our understanding of cost and technical requirements before program initiation; and (2) requiring a Milestone Decision Authority-conducted post-Critical Design Review Assessment in order to ensure that programs are proceeding through the System Development and Demonstration phase consistent with sound engineering practice and the approved acquisition program baseline. Together we believe these new or pending policies will enhance existing policy, and substantively contribute to improved cost estimates, better informed decision making and shorter cycle times. Recommendation 2 includes a provision that all new development programs have "manageable development cycles." This conclusion is based on the commercial product development models reviewed by GAO and mentioned in the report, specifically those from Eli Lilly, IBM, and Motorola. While we generally agree that DoD product development cycles should be "manageable," these development cycles may bear little relation to those of the three commercial firms cited by GAO in the report. The use by GAO of three commercial firm product development models as a basis for determining `best practice' in DoD has several flaws. First, DoD as a public sector institution, in contrast to a private sector firm, has key differences in goals and incentives. All of the DoD weapon system programs cited in the report are developed through contracts from the public to the private sector, which contain significantly different incentives and provisions than the product development programs typically undertaken internally by firms in the private sector. It may be more appropriate that instead of benchmarking DoD against private sector firms, GAO should be benchmarking DoD against best practices in the public sector throughout the world. Further the commodity classes for comparison to DoD development programs for major weapon systems that GAO selected, we believe, are not a good comparison. The three firms cited in the report are in the pharmaceutical, computer products and services, and the radio and cellular telephony sectors. Yet the implication of the report is that the development cycles in these product sectors can be directly applied to systems like the Joint Strike Fighter program. At a minimum, GAO might better examine commercial product developments in comparable commodity sectors, such as the commercial aircraft sector, to avoid drawing conclusions that may be invalid based on analogies to dissimilar commodity sectors. In order to more thoroughly review the study, the DoD requested that the GAO share its methodology, raw data, interview notes, detailed analyses, working papers, etc. Although to date the GAO has not provided the aforementioned materials, the Department would be pleased to work with GAO to better understand the details of such a comparison. Recommendation 3: The GAO recommends that the Secretary of Defense require all cost estimates submitted for funding a program at Milestone decisions to be reported as a range of likely costs and the associated levels of risk and uncertainty. At Milestone A, require estimates that allow for a wide range of likely costs. At Milestone B, require estimates that, based on knowledge gained, are more precise--in line with best practice standards. (p. 28/GAO Draft Report) DOD Response: Non-concur. While we and the DoD Cost Analysis Improvement Group (CRAIG) agree with the spirit of the recommendation to consider risks at key milestones in development programs; however the GAO recommendation implies that the DoD should calculate a range of costs and associated confidence levels using Monte Carlo analysis techniques. The CAIG has observed limitations to this analysis methodology when used in practice: Most mathematical implementations of this methodology assume no correlation of work breakdown structure elements in developing distributions of potential cost outcomes. In most cases this simplifying assumption is incorrect, particularly for cost estimates developed for product-oriented work breakdown structures, as are commonly used in the DoD. This assumption results in calculated distributions of potential cost outcomes that are "too narrow" relative to the actual distribution of cost outcomes, providing misleading results. The CAIG does routinely calculate cost risks for major milestone reviews by examining the key structural elements of cost estimates, including schedule durations, technical risks, contract vehicles, incentives, and management structures. In certain situations, the CAIG has reported a range of life-cycle costs because of the poor quality of the definition of the program to be undertaken in the Cost Analysis Requirements Description document. For example, the CAIG reported a likely range of Future Combat Systems (FCS) development costs on several occasions, a fact that GAO did not include in its report. In summary, the CAIG tailors the risk analysis to the specific program and milestone at hand, calculating and presenting ranges of costs when it is appropriate to do so. The specification. of a single methodology and practice to be used at each and every milestone review of a Major Defense Acquisition Program is not appropriate. In the GAO report, Table 1, Development Costs Estimates and Baselines for 20 Major Weapon System Programs, contains information on development cost estimates and cost baselines. Unfortunately, the report does not consider causal factors bearing on the estimates: (1) methodological issues, (2) incomplete, errorful, or volatile requirements or program specification, (3) engineering change proposals, (4) Congressionally mandated changes in program funding levels or unit quantities, etc. Additionally, the report does not specify the metrics whereby a cost estimate is considered as acceptable. In order to assist in maintaining an awareness of overall risk with its acquisition programs, the Air Force is evaluating the idea of setting a goal of annual updates to program cost estimates for major defense acquisition programs, versus the DoD 5000 series requirement to update for each milestone review. This should bring newly-acquired knowledge to bear more quickly and bring problems to light in a more timely manner. [End of section] Related GAO Products: Defense Acquisitions: Assessments of Selected Weapon Programs. [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-08-467SP]. Washington, D.C.: March 31, 2008. Best Practices: Increased Focus on Requirements and Oversight Needed to Improve DOD's Acquisition Environment and Weapon System Quality. [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-08-294], Washington, D.C.: February 1, 2008. Cost Assessment Guide: Best Practices for Estimating and Managing Program Costs. [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07- 1134SP], Washington, D.C.: July 2007. Defense Acquisitions: Realistic Business Cases Needed to Execute Navy Shipbuilding Programs. [hyperlink, http://www.gao.gov/cgi- bin/getrpt?GAO-07-943T], Washington, D.C.: July 24, 2007. Best Practices: An Integrated Portfolio Management Approach to Weapon System Investments Could Improve DOD's Acquisition Outcomes. [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-388], Washington, D.C.: March 30, 2007. Space Acquisitions: DOD Needs to Take More Action to Address Unrealistic Initial Cost Estimates of Space Systems. [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-96], Washington, D.C.: November 17, 2006. Defense Acquisitions: Major Weapon Systems Continue to Experience Cost and Schedule Problems under DOD's Revised Policy. [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-06-368]. Washington, D.C.: April 13, 2006. Defense Acquisitions: Actions Needed to Get Better Results on Weapons Systems Investments. [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO- 06-585T]. Washington, D.C.: April 5, 2006. Best Practices: Better Support of Weapon System Program Managers Needed to Improve Outcomes. [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO- 06-110]. Washington, D.C.: November 30, 2005. Future Years Defense Program: Actions Needed to Improve Transparency of DOD's Projected Resource Needs. [hyperlink, http://www.gao.gov/cgi- bin/getrpt?GAO-04-514]. Washington, D.C.: May 7, 2004. Best Practices: Capturing Design and Manufacturing Knowledge Early Improves Acquisition Outcomes. [hyperlink, http://www.gao.gov/cgi- bin/getrpt?GAO-02-701]. Washington, D.C.: July 15, 2002. Best Practices: Better Matching of Needs and Resources Will Lead to Better Weapon System Outcomes. [hyperlink, http://www.gao.gov/cgi- bin/getrpt?GAO-01-288]. Washington, D.C.: March 8, 2001. Footnotes: [1] GAO, Best Practices: An Integrated Portfolio Management Approach to Weapon System Investments Could Improve DOD's Acquisition Outcomes. [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-388], Washington, D.C.: March 30, 2007. [2] Global Hawk, an Air Force unmanned aircraft system, is intended to provide intelligence, surveillance, and reconnaissance capabilities; JSF is a joint Air Force, Navy, and Marine Corps program to develop and field stealthy fighter aircraft to replace DOD's aging fighter and attack aircraft; FCS is an Army program intended to provide advanced, networked combat and sustainment systems, unmanned ground and air vehicles, and unattended sensors and munitions; WIN-T is intended to provide the Army with a high-speed, high-capacity communications network; MMA is a Navy program intended to provide persistent antisubmarine and antisurface warfare, and intelligence, surveillance, and reconnaissance capabilities. [3] GAO, Cost Assessment Guide: Best Practices for Estimating and Managing Program Costs. [hyperlink, http://www.gao.gov/cgi- bin/getrpt?GAO-07-1134SP] (Washington, D.C.: July 2007). [4] Full funding is a DOD 5000.2 requirement for formal program initiation of an acquisition program. In this sense, full funding means having an approved current (and projected) resource stream to execute the acquisition program; that is, program funding is included both in the budget and in the out-years of the Future Years Defense Program sufficient to cover the current and future efforts described in the acquisition strategy. [5] DOD modified the PPBS process, renamed the PPBE, in 2003. PPBE in its entirety is not implemented every year even though DOD must request funding from congress annually. Full-scale planning and programming activities are conducted in even-numbered years (called on-years) while programming adjustments are made in odd-numbered years (called off- years). As a result of this modification, the number of years covered by the FYDP now alternates between 5 years (the President's budget plus 4 years) and 6 years (the President's budget plus 5 years). [6] The Milestone Decision Authority for all Major Defense Acquisition Programs is the Under Secretary of Defense for Acquisition, Technology & Logistics unless delegated to the head of a DOD component. [7] Although DOD policy requires full funding within the FYDP, most major weapon system acquisition programs have development cycles that extend beyond the FYDP time period. However, there is no requirement to constrain development cycles to fit within that same time period. [8] U.S.C. Title 10 § 2366(a), which was enacted in 2006, specifies that a major defense acquisition program may not receive Milestone B approval, or Key Decision Point B approval in the case of a space program, until the Milestone decision authority certifies that--(1) the program technology has been demonstrated in a relevant environment; (2) the program demonstrates a high likelihood of accomplishing its mission; (3) the program is affordable when considering the per unit cost and the total acquisition cost in the context of the total resources available during the period covered by the future-years defense program submitted during the fiscal year in which the certification is made; (4) the department has completed an analysis of alternatives; (5) the program is affordable when considering the ability of DOD to accomplish the program's mission using alternative systems; (6) the Joint Requirements Oversight Council has accomplished its duties with respect to the program pursuant to section 181 (b) of this title, including an analysis of the operational requirements for the program; and (7) the program complies with all relevant policies, regulations, and directives of DOD. [9] Programs are not required to hold a Milestone A review, the point at which concept refinement ends and technology development begins. However, in 2007, Congress enacted legislation, which specifies that a major defense acquisition program beginning after March 2008 may not receive Milestone A approval, to begin a technology development program, until the Milestone Decision Authority certifies to Congress that (1) the system fulfills an approved initial capabilities document; (2) the system is being executed by an entity with a relevant core competency as identified by the Secretary of Defense; (3) if the system duplicates a capability already provided by an existing system, the duplication provided by such system is necessary and appropriate; and (4) a cost estimate for the system has been submitted. DOD is currently revising its policy and guidance for conducting and certifying Milestone A reviews. [10] See Related GAO Products at the end of this report. [11] The funding needed within the FYDP for each program was calculated using the funding data contained in the RDT&E Annual Funding Summary of the first SAR issued after system development began. The amount of funding established in the FYDP was calculated using each program's RDT&E budget justification documentation for the year in which system development was initiated. [12] DOD has a formal process for reprogramming appropriated funds, including if necessary, congressional notification and approval. [13] GAO, Space Acquisitions: DOD Needs to Take More Action to Address Unrealistic Initial Cost Estimates for Space Systems. [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-96], Washington, D.C.: November 17, 2006. [14] [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-388]. [15] [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-388]. [16] Assessment Panel of the Defense Acquisition Performance Assessment Project for the Deputy Secretary of Defense, Defense Acquisition Performance Assessment Report (January 2006); Defense Science Board Summer Study on Transformation: A Progress Assessment (February 2006); RAND, Historical Cost Growth of Completed Weapon System Programs (2006); Institute for Defense Analysis, Costs Growth in Major Weapon Procurement Programs, Presentation (2005). [17] Defense Science Board, Report of the Defense Science Board /Air Force Advisory Board, Joint Task Force on Acquisition of National Security Space Programs, May 2003 [18] Systems engineering is a technical management tool that provides the knowledge necessary to translate requirements into specific, achievable capabilities. By using the tools of systems engineering during the early phases of concept refinement and technology development, acquisition decision makers and developers can work together to close gaps between requirements and available resources well before system development starts. [19] An AOA should compare the costs and benefits of alternative solutions for meeting a validated need. As such an AOA should assess the life-cycle cost, schedule, and operational effectiveness of each possible solution and identify a preferred alternative. [20] GAO, Defense Acquisitions: Assessments of Selected Major Weapon Programs. [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-08-467SP]. Washington, D.C.: March 31, 2008; Defense Acquisitions: Major Weapon Systems Continue to Experience Cost and Schedule Problems under DOD's Revised Policy. [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-06- 368]. Washington, D.C.: April 13, 2006. [21] [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-06-368] and [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-388]. [22] Institute for Defense Analysis, Future Combat Systems (FCS) Cost Review: Summary of Findings, P-4212, April 2007, p. S-2. [23] GAO, Tactical Aircraft: Opportunity to Reduce Risks in the Joint Strike Fighter Program with Different Acquisition Strategy. [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-05-271] (Washington, D.C.: March 15, 2005). [24] GAO, Space Acquisitions: DOD Needs to Take More Action to Address Unrealistic Initial Cost Estimates for Space Systems. [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-96] (Washington, D.C.: Nov. 17, 2006). [25] GAO, Defense Acquisitions: Realistic Business Cases Needed to Execute Navy Shipbuilding Programs. [hyperlink, http://www.gao.gov/cgi- bin/getrpt?GAO-07-943T], Washington, D.C.: July 24, 2007. [26] For management to make good decisions about programs, our Cost Assessment Guide calls for a quantitative risk and uncertainty analysis to assess variability in an estimate. [27] A Defense Science Board report recommended funding space acquisition programs at the 80-percent confidence level (Report of the Defense Science Board/Air Force Advisory Board, Joint Task Force on Acquisition of National Security Space Programs, May 2003); and subsequently the Air Force issued guidance in March 2007 stating that an 80-percent confidence level is the objective for space programs, but it is too soon to assess whether this policy has resulted in more realistic estimates. [28] GAO, Future Years Defense Program: Risks in Operation and Maintenance and Procurement Programs. [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-01-33] (Washington, D.C.: October 5, 2000); Best Practices: An Integrated Portfolio Management Approach to Weapon System Investments Could Improve DOD's Acquisition Outcomes. [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-388] (Washington, D.C.: March 30, 2007). [29] GAO, Best Practices: Better Support of Weapon System Program Managers Needed to Improve Outcomes, [hyperlink, http://www.gao.gov/cgi- bin/getrpt?GAO-06-110] (Washington, D.C.: November 30, 2005). [30] See Related GAO Products at the end of this report. [31] IBM's contracts are not focused solely on program costs; they also establish goals, objectives, and allowable deviations for other program measures such as schedule and revenue. [32] Assessment Panel of the Defense Acquisition Performance Assessment Project for the Deputy Secretary of Defense, Defense Acquisition Performance Assessment Report (Jan. 2006). [33] Defense Acquisitions: Assessments of Selected Major Weapon Programs. [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-08-467SP]. Washington, D.C.: March 31, 2008. [34] Global Hawk, an Air Force unmanned aircraft system, is intended to provide intelligence, surveillance, and reconnaissance capabilities; JSF is a joint Air Force, Navy, and Marine Corps program to develop and field stealthy fighter aircraft to replace DOD's aging fighter and attack aircraft; FCS is an Army program intended to provide advanced, networked combat and sustainment systems, unmanned ground and air vehicles, and unattended sensors and munitions; WIN-T is intended to provide the Army with a high-speed, high-capacity communications network; MMA is a Navy program intended to provide persistent antisubmarine and antisurface warfare, and intelligence, surveillance, and reconnaissance capabilities. [35] Eli Lilly is one of the largest corporations in the world and engages in pharmaceutical research and development around the world; IBM is the largest supplier of hardware, software, and information technology services; and Motorola is a Fortune 100 global communications leader. We also contacted the other two companies that provided input to our prior review--Proctor & Gamble and Caterpillar-- but relevant officials were not available to meet with us for follow-up discussions during the timeframes of our review. [End of section] GAO's Mission: The Government Accountability Office, the audit, evaluation and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each weekday, GAO posts newly released reports, testimony, and correspondence on its Web site. To have GAO e-mail you a list of newly posted products every afternoon, go to [hyperlink, http://www.gao.gov] and select "E-mail Updates." Order by Mail or Phone: The first copy of each printed report is free. Additional copies are $2 each. A check or money order should be made out to the Superintendent of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a single address are discounted 25 percent. Orders should be sent to: U.S. Government Accountability Office: 441 G Street NW, Room LM: Washington, D.C. 20548: To order by Phone: Voice: (202) 512-6000: TDD: (202) 512-2537: Fax: (202) 512-6061: To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: E-mail: fraudnet@gao.gov: Automated answering system: (800) 424-5454 or (202) 512-7470: Congressional Relations: Ralph Dawn, Managing Director, dawnr@gao.gov: (202) 512-4400: U.S. Government Accountability Office: 441 G Street NW, Room 7125: Washington, D.C. 20548: Public Affairs: Chuck Young, Managing Director, youngc1@gao.gov: (202) 512-4800: U.S. Government Accountability Office: 441 G Street NW, Room 7149: Washington, D.C. 20548: