This is the accessible text file for GAO report number GAO-13-826R entitled 'Department of Defense's Waiver of Competitive Prototyping Requirement for the VXX Presidential Helicopter Replacement Program' which was released on September 6, 2013. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. GAO-13-826R: GAO: United States Government Accountability Office: 441 G St. N.W. Washington, DC 20548: September 6, 2013: Congressional Committees: Department of Defense's Waiver of Competitive Prototyping Requirement for the VXX Presidential Helicopter Replacement Program: The Weapon Systems Acquisition Reform Act of 2009, as amended (WSARA), requires the Secretary of Defense to modify guidance to ensure that the acquisition strategy for each major defense acquisition program provides for competitive prototypes before Milestone B approval--which authorizes entry into system development--unless the Milestone Decision Authority waives the requirement.[Footnote 1] Competitive prototyping, which involves commercial, government, or academic sources producing early prototypes of weapon systems or critical subsystems, can help Department of Defense (DOD) programs reduce technical risk, refine requirements, validate designs and cost estimates, and evaluate manufacturing processes prior to making major commitments of resources. It can also help reduce the time it takes to field a system, and as a result, reduce its acquisition cost. WSARA states that the Milestone Decision Authority may waive the competitive prototyping requirement only on the basis that (1) the cost of producing competitive prototypes exceeds the expected life-cycle benefits (in constant dollars) of producing such prototypes, including the benefits of improved performance and increased technological and design maturity that may be achieved through competitive prototyping; or (2) but for such a waiver, DOD would be unable to meet critical national security objectives. WSARA also provides that whenever a Milestone Decision Authority authorizes a waiver of the competitive prototyping requirement on the basis of what WSARA describes as "excessive cost," the Milestone Decision Authority is required to submit notification of the waiver, together with the rationale, to the Comptroller General of the United States at the same time it is submitted to the congressional defense committees. WSARA further provides that no later than 60 days after receipt of a notification of a waiver, we are mandated to review the rationale for the waiver and submit a written assessment of that rationale to the congressional defense committees.[Footnote 2] On July 10, 2013, we received notice from DOD that it had waived the competitive prototyping requirement for the VXX Presidential Helicopter Replacement Program. The Navy's VXX program is an effort to replace the aging fleet of 19 VH-3D and VH-60N helicopters used to transport the President, Vice President, heads of state, and others. The program is the Navy's second effort to replace these helicopters. The first effort, the VH-71 program, was canceled in 2009 due to significant cost growth, schedule delays, and performance issues. Since then, the Navy has reduced the helicopter's passenger capacity, range, transportability, operational availability, airspeed, and hover performance requirements. These performance trade-offs allowed the Navy to adopt a streamlined acquisition approach that involves integrating mature communications and mission systems into an existing commercial or military helicopter that is already in production without the need for a technology development phase. The Navy expects this approach to lower the program's cost, shorten its schedule, and reduce its risk. In this report, we assess DOD's rationale for waiving the competitive prototyping requirement for the VXX program and the analysis used to support it. To conduct our assessment, we compared the rationale in the waiver to the WSARA requirement to determine the extent to which the waiver is consistent with the statute. In addition, we reviewed the Navy's cost-benefit analysis, which provides the data and assumptions on which the waiver is based. We also questioned DOD, Navy, and VXX program officials to clarify information in this documentation, as necessary. Further, we relied on our April 2013 report on the VXX program for information on the degree to which the cost estimates used to support the Navy's cost-benefit analysis complied with cost estimating and assessment best practices.[Footnote 3] We did not assess the estimated cost benefits of the various prototyping scenarios in the Navy's cost-benefit analysis against these best practices criteria, but did take other steps to assess their reasonableness. We conducted this performance audit from July 2013 to September 2013 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Results in Brief: DOD's rationale for waiving WSARA's competitive prototyping requirement for VXX addresses one of the two bases provided in the statute; namely that the cost of producing competitive prototypes exceeds the expected life-cycle benefits (in constant dollars) of producing the prototypes. The VXX program's acquisition strategy provided the primary justification for the prototyping waiver. According to the waiver, VXX requirements can be met by integrating an existing, in-production, flight-proven aircraft with mature mission systems. The Navy in its waiver request also concluded that the integration activities planned for the VXX program do not require additional technology maturation or risk reduction beyond that already being accomplished by the government through its own prototyping of certain critical mission subsystems. Recognizing that the intent of competitive prototyping is to reduce cost and risk, DOD has taken other actions that could arguably achieve these goals. Specifically, DOD decided to reduce requirements, use an existing aircraft, and mature critical subsystems before integrating them on the aircraft. In the waiver, DOD also found reasonable the Navy's cost-benefit analysis, which examined multiple acquisition strategies with system- and subsystem-level prototyping from one or two contractors. In all, the Navy examined six different acquisition strategies and concluded that requiring competitive prototyping would delay fielding an initial operational capability by 16 months and increase development costs by about $782 million to $3.38 billion (in base year 2011 dollars), depending on the type and number of prototypes. The Navy also estimated that the more costly system-level prototyping strategies could achieve an estimated $542 million (in base year 2011 dollars) in life-cycle cost benefits by improving the reliability of the aircraft, which in turn could reduce the number of helicopters required. According to the Navy, the cost data used in its cost-benefit analysis were drawn from the VXX analysis of alternatives study and related activities. In April 2013, we found that the Navy used cost estimating procedures on the VXX analysis of alternatives that substantially complied with best practices. Waiver Rationale Is Consistent with WSARA and Supporting Analysis Incorporated Lessons Learned from Prior Waivers: DOD's rationale for waiving WSARA's competitive prototyping requirement for VXX addressed one of the two bases provided for a waiver in the statute, namely that the cost of producing competitive prototypes exceeds the expected life-cycle benefits, including the benefits of improved performance and increased technological and design maturity that may be achieved through competitive prototyping. In the waiver, DOD also stated that the cost of producing a single prototype of the VXX or its critical subsystems before Milestone B exceeded the expected life-cycle benefits.[Footnote 4] The VXX program's acquisition strategy provided the primary justification for the prototyping waiver. According to the waiver, VXX requirements can be met by integrating an existing, in-production, flight-proven aircraft with mature mission systems. The program is conducting government-led prototyping of certain critical mission subsystems, including an Internet Protocol-based communication capability that is comprised of existing commercial equipment. We have previously found that there are potential risks associated with integrating government- developed mission systems into a new airframe.[Footnote 5] The Navy's waiver request acknowledged that integration could be a challenge, but it concluded that the integration activities planned for the VXX program do not require technology maturation or risk reduction beyond that already being accomplished by the government. Therefore, it concluded that the program would not benefit from additional prototyping at the system or subsystem level. Recognizing that the intent of competitive prototyping is to reduce cost and risk, DOD has taken other actions that could arguably achieve these goals. Specifically, DOD decided to reduce requirements, use an existing aircraft, and mature critical subsystems before integrating them on the aircraft. In the waiver, DOD found reasonable the Navy's cost-benefit analysis, which examined multiple acquisition strategies with system-and subsystem-level prototyping from one or two contractors. The Navy's approach was consistent with a key principle in DOD's policy on economic analysis, which states that each feasible alternative for meeting an objective must be considered, and its life-cycle costs and benefits evaluated.[Footnote 6] In all, the Navy examined six different acquisition strategies with various degrees of prototyping and compared the development cost of the VXX as well as the potential life-cycle benefits of prototyping under each scenario. Based on this analysis, DOD concluded that requiring competitive prototyping of critical subsystems would delay fielding an initial operational capability by 16 months and increase development costs by about $1.189 billion for two industry prototypes and $782 million for a single prototype in base year 2011 dollars. The waiver also concluded that pursuing an aircraft prototyping strategy to improve aircraft performance would increase development costs by about $3.38 billion for a strategy with two industry prototypes and $2.84 billion for a single prototype in base year 2011 dollars. Table 1 includes a description of each of these acquisition strategies and their total estimated development cost, schedule, and expected benefits. Table 1: Navy's Estimated Cost, Schedule, and Benefits for Various VXX Acquisition Strategies: Acquisition strategy: Planned strategy with government prototyping of critical mission subsystems and no aircraft prototyping[A]; Total development cost including prototyping (in base year 2011 billions): $4.33-4.51 billion; Time to field initial capability: 85 months; Expected benefits of prototyping: [Empty]. Subsystem prototypes: Acquisition strategy: One contractor prototyping critical mission subsystems; Total development cost including prototyping (in base year 2011 billions): $5.29 billion; Time to field initial capability: 101 months; Expected benefits of prototyping: None. Acquisition strategy: Two contractors prototyping critical mission subsystems; Total development cost including prototyping (in base year 2011 billions): $5.70 billion; Time to field initial capability: 101 months; Expected benefits of prototyping: None. Aircraft prototypes[B]: Acquisition strategy: Major development program with one aircraft prototype; Total development cost including prototyping (in base year 2011 billions): $7.17 billion; Time to field initial capability: At least 101 months; Expected benefits of prototyping: Life cycle cost savings of $542 million (base year 2011 dollars) driven by improved operational availability and reduction in number of helicopters required; potential capability improvements. Acquisition strategy: Major development program with two competing aircraft prototypes; Total development cost including prototyping (in base year 2011 billions): $7.71 billion; Time to field initial capability: At least 101 months; Expected benefits of prototyping: Life cycle cost savings of $542 million (base year 2011 dollars) driven by improved operational availability and reduction in number of helicopters required; potential capability improvements. Source: GAO analysis of Navy's VXX cost-benefit analysis. [A] This strategy was treated as two separate scenarios in the Navy's cost-benefit analysis. [B] Assumes government prototyping of critical mission subsystems. [End of table] The Navy used cost estimating procedures on the VXX competitive prototyping cost-benefit analysis that were substantially compliant with best practices.[Footnote 7] According to the Navy, the cost data used in its cost-benefit analysis were drawn from the VXX analysis of alternatives study and related activities. This analysis was completed from 2010 to 2012 and provided the basis for the program's current acquisition strategy. In our April 2013 report, we found that the cost estimating procedure used for the analysis of alternatives was substantially compliant with best practices criteria.[Footnote 8] Specifically, we found the analysis of alternatives cost estimate to be comprehensive and well documented. We also found that it was substantially accurate and partially met our criteria for being credible. The cost estimating best practices associated with each of those characteristics used in assessing the analysis of alternatives are provided in enclosure 1. The Navy incorporated lessons learned from DOD's first two competitive waivers to improve certain aspects of the VXX cost-benefit analysis. In an August 2012 report, we concluded that the rationale for the Enhanced Polar System program waiver was consistent with WSARA, but the cost-benefit analysis supporting it was incomplete.[Footnote 9] That analysis did not include a specific dollar value or range of values for the estimated life-cycle benefits of competitive prototyping. We also found that the Air Force's estimate of the cost of competitive prototyping was prepared by the program office and not independently estimated or reviewed by other DOD cost estimating organizations. In a March 2013 report, we found that the Air Force only evaluated one potential approach to implementing competitive prototyping in its waiver request for the Combat Rescue Helicopter program.[Footnote 10] We concluded that the Air Force could have more fully evaluated the cost and benefits of additional prototyping strategies, including more limited approaches that are mentioned in WSARA or DOD's implementing memorandum, such as producing a single prototype or prototyping critical subsystems before Milestone B. The VXX cost-benefit analysis addressed all of these points. Agency Comments and Our Evaluation: We provided a draft of this report to DOD for comment. In its written comments, which are reprinted in enclosure 2, DOD concurred with our assessment of the VXX prototyping waiver. We are sending copies of this report to interested congressional committees, the Secretary of Defense, the Secretary of the Navy, and the Commandant of the Marine Corps. In addition, the report will be available at no charge on the GAO website at [hyperlink, http://www.gao.gov]. If you or your staff have any questions, please contact me at (202) 512-4841 or sullivanm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report were Ron Schwenn, Assistant Director; Bruce Thomas, Assistant Director; Marie P. Ahearn; Pedro Almoguera; Jerry Clark; Bonita Oden; Kenneth E. Patton; and Carol Petersen. Signed by: Michael J. Sullivan: Director, Acquisition and Sourcing Management: Enclosures - 2: List of Committees: The Honorable Carl Levin: Chairman: The Honorable James Inhofe: Ranking Member: Committee on Armed Services: United States Senate: The Honorable Richard J. Durbin: Chairman: The Honorable Thad Cochran: Ranking Member: Subcommittee on Defense: Committee on Appropriations: United States Senate: The Honorable Howard P. "Buck" McKeon: Chairman: The Honorable Adam Smith: Ranking Member: Committee on Armed Services: House of Representatives: The Honorable C.W. Bill Young: Chairman: The Honorable Pete Visclosky: Ranking Member: Subcommittee on Defense: Committee on Appropriations: House of Representatives: [End of section] Enclosure 1: Cost Estimating Best Practices: Characteristic: Comprehensive; Best practices: * The cost estimate includes all life cycle costs; * The cost estimate completely defines the program, reflects the current schedule, and is technically reasonable; * The cost estimate work breakdown structure (WBS) is product- oriented, traceable to the statement of work/objective, and at an appropriate level of detail to ensure that cost elements are neither omitted nor double-counted; * The estimate documents all cost-influencing ground rules and assumptions. Characteristic: Well documented; Best practices: * The documentation should capture the source data used, the reliability of the data, and how the data were normalized; * The documentation describes in sufficient detail the calculations performed and the estimating methodology used to derive each element's cost; * The documentation describes step by step how the estimate was developed so that a cost analyst unfamiliar with the program could understand what was done and replicate it; * The documentation discusses the technical baseline description and the data in the baseline is consistent with the estimate; * The documentation provides evidence that the cost estimate was reviewed and accepted by management. Characteristic: Accurate; Best practices: * The cost estimate results are unbiased, not overly conservative or optimistic and based on an assessment of most likely costs; * The estimate has been adjusted properly for inflation; * The estimate contains few, if any, minor mistakes; * The cost estimate is regularly updated to reflect significant changes in the program so that it is always reflecting current status. Variances between planned and actual costs are documented, explained, and reviewed; * The estimate is based on a historical record of cost estimating and actual experiences from other comparable programs. Characteristic: Credible; Best practices: * The cost estimate includes a sensitivity analysis that identifies a range of possible costs based on varying major assumptions, parameters, and data inputs; * A risk and uncertainty analysis was conducted that quantified the imperfectly understood risks and identified the effects of changing key cost driver assumptions and factors; * Major cost elements were cross-checked to see whether results were similar; * An independent cost estimate was conducted by a group outside the acquiring organization to determine whether other estimating methods produce similar results. Source: GAO. [End of table] [End of section] Enclosure 2: Comments from the Department of Defense: Assistant Secretary of Defense: Acquisitions: 3015 Defense Pentagon: Washington, DC 20301-3015: August 30, 2013: Mr. Michael Sullivan: Director, Acquisition and Sourcing Management: U.S. Government Accountability Office: 441 G Street, NW: Washington, DC 20548: Dear Mr. Sullivan: This is the Department of Defense response to the Government Accountability Office (GAO) Draft Report, GAO-13-862R, "Competitive Prototyping: Department of Defense's Waiver of the Competitive Prototyping Requirement for the VXX Presidential Helicopter Replacement Program" dated August 22, 2013 (GAO Code 121169). The Department appreciates the effort of the GAO and the opportunity to comment on the draft report. The Department concurs with and appreciates GAO's conclusions that the VXX waiver rationale is consistent with Weapons Systems Acquisition Reform Act of 2009, Public Law 11123 and that supporting analysis incorporated lessons learned from prior waivers. We have no additional comments on the draft report. Signed by: Katrina McFarland: [End of section] Footnotes: [1] Pub. L. No. 111-23, § 203(a), as amended by the Ike Skelton National Defense Authorization Act for Fiscal Year 2011, Pub. L. No. 111-383, § 813. DOD modified its guidance related to the operation of its acquisition system through Directive-Type Memorandum (DTM) 09-027, "Implementation of Weapon System Acquisition Reform Act of 2009 (Dec. 4, 2009, incorporating Change 3, Dec. 9, 2011). Major defense acquisition programs are those estimated by DOD to require an eventual total expenditure for research, development, test, and evaluation of more than $365 million, or for procurement, including all planned increments or spirals, of more than $2.19 billion in fiscal year 2000 constant dollars. The Milestone Decision Authority for major defense acquisition programs is the Under Secretary of Defense for Acquisition, Technology, and Logistics; the head of a DOD component; or if delegated the component acquisition executive. [2] Pub. L. No. 111-23 § 203(b)(1) & (2). [3] GAO, Presidential Helicopter Acquisition: Program Makes Progress in Balancing Requirements, Costs, and Schedule, [hyperlink, http://www.gao.gov/products/GAO-13-257] (Washington, D.C.: Apr. 9, 2013). [4] WSARA provides that whenever a Milestone Decision Authority authorizes a competitive prototyping waiver, the program is required to produce a prototype prior to Milestone B approval if the expected life-cycle benefits (in constant dollars) of producing such prototype exceed its cost and its production is consistent with achieving critical national security objectives. Pub. L. No. 111-23 § 203(a)(3)(A). [5] [hyperlink, http://www.gao.gov/products/GAO-13-257]. [6] DOD Instruction 7041.3, Economic Analysis for Decisionmaking (Nov. 7, 1995). [7] GAO, GAO Cost Estimating and Assessment Guide: Best Practices for Developing and Managing Capital Program Costs, [hyperlink, http://www.gao.gov/products/GAO-09-3SP] (Washington, D.C.: Mar. 2, 2009). [8] GAO-13-257. [9] GAO, Department of Defense's Waiver of Competitive Prototyping Requirements for Enhanced Polar System Program, [hyperlink, http://www.gao.gov/products/GAO-12-983R] (Washington, D.C.: Aug. 23, 2012). [10] GAO, Department of Defense's Waiver of Competitive Prototyping Requirements for Combat Rescue Helicopter Program, [hyperlink, http://www.gao.gov/products/GAO-13-313R] (Washington, D.C.: Mar. 7, 2013). [End of correspondence]