This is the accessible text file for GAO report number GAO-09-303 entitled 'Joint Strike Fighter: Accelerating Procurement before Completing Development Increases the Government's Financial Risk' which was released on March 12, 2009. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. Report to Congressional Committees: United States Government Accountability Office: GAO: March 2009: Joint Strike FighteR: Accelerating Procurement before Completing Development Increases the Government's Financial Risk: Joint Strike Fighter: GAO-09-303: GAO Highlights: Highlights of GAO-09-303, a report to congressional committees. Why GAO Did This Study: The Joint Strike Fighter (JSF) is the Department of Defense’s (DOD) most complex and ambitious aircraft acquisition, seeking to simultaneously produce and field three different versions of the aircraft for the Air Force, Navy, Marine Corps, and eight international partners. The total investment required now exceeds $1 trillion—more than $300 billion to acquire 2,456 aircraft and $760 billion in life cycle operating and support costs, according to program estimates. The Ronald W. Reagan National Defense Authorization Act for Fiscal Year 2005 requires GAO to review the JSF program annually for 5 years. This is the fifth and final report under the mandate in which GAO (1) determines the program’s progress in meeting cost, schedule, and performance goals; (2) assesses manufacturing results and schedule risks; and (3) evaluates development test plans, progress, and risks. GAO’s work included analyses of a wide range of program documents, cost data and interviews with defense and contractor officials. What GAO Found: JSF development will cost more and take longer than reported to the Congress last year, and DOD wants to accelerate procurement. Two recent estimates project additional costs ranging from $2.4 billion to $7.4 billion and 1 to 3 more years to complete development. Despite cost and schedule troubles, DOD wants to accelerate JSF procurement by 169 aircraft from fiscal years 2010 through 2015; this could require up to $33.4 billion in additional procurement funding for those 6 years. DOD plans to procure hundreds of aircraft on cost-reimbursement contracts, magnifying the financial risk to the government. Ongoing manufacturing inefficiencies and parts problems have significantly delayed the delivery of test assets. The prime contractor has extended manufacturing schedules three times and delivered 2 of 13 test aircraft. The program is still recovering from earlier problems that resulted in design changes, late parts deliveries, and inefficient manufacturing. The contractor is taking positive steps to improve operations, the supplier base, and schedule management. Schedule risk analyses could further enhance management insight into problem areas and inform corrective actions. Officials expect to deliver all test aircraft and fix many problems by 2010. By then, DOD plans to have purchased 62 operational aircraft and will be ramping up procurement. Procuring large numbers of production jets while still working to deliver test jets and mature manufacturing processes does not seem prudent, and looming plans to accelerate procurement will be difficult to achieve cost effectively. DOD’s revised test plan adds a year to the schedule, better aligns resources and availability dates, and lessens the overlap between development and operational testing, but it still allows little time for error discovery and rework. DOD’s decision late in 2007 to reduce test aircraft and flight tests adds to risks while any additional delays in delivering test aircraft will further compress the schedule. The revised plan relies on state-of-the-art simulation labs, a flying test bed, and desk studies to verify nearly 83 percent of JSF capabilities. Only 17 percent is to be verified through flight testing. Despite advances, the ability to so extensively substitute for flight testing has not yet been demonstrated. Significant overlap of development, test, and procurement results in DOD making substantial investments before flight testing proves that the JSF will perform as expected. Under the accelerated procurement plan, DOD may procure 360 aircraft costing an estimated $57 billion before completing development flight testing. Table: Procurement Investments and Progress of Flight Testing: Cumulative procurement (billions of dollars); 2007: $0.9; 2008: $3.6; 2009: $7.4; 2010: $15.4; 2011: $28.2; 2012: $42.5; 2013: $57.0; 2014: $72.3; 2015: $89.0. Cumulative aircraft procured; 2008: 14; 2009: 30; 2010: 62; 2011: 132; 2012: 241; 2013: 360; 2014: 506; 2015: 684. Percent flight tests completed; 2007: <1%; 2008: <1%; 2009: 2%; 2010: 9%; 2011: 34%; 2012: 62%; 2013: 88%; 2014: 100%; 2015: -. Source: GAO analysis of DOD data. [End of table] What GAO Recommends: GAO recommends that DOD (1) report to the congressional defense committees on the risks and mitigation strategy for use of cost reimbursement contracts for procurement and plans to transition to fixed price contracts and (2) ensure that the prime contractor performs periodic schedule risk analyses to improve schedule and budget actions. DOD agreed to take these actions. To view the full product, including the scope and methodology, click on [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-09-303]. For more information, contact Michael J. Sullivan at (202) 512-4841 or sullivanm@gao.gov. [End of section] Contents: Letter: Results in Brief: Background: More Money and Time Will Be Needed to Complete JSF Development, While DOD Plans to Accelerate Procurement: Continued Manufacturing Inefficiencies Will Make It Difficult for the Program to Meet Its Production Schedule: The JSF's Test Plan Is Improved but Faces Numerous Challenges to Complete Development on Time and on Budget: Conclusions: Recommendations for Executive Action: Agency Comments and Our Evaluation: Appendix I: Scope and Methodology: Appendix II: GAO Assessment of Prime Contractor Schedule Management Processes: Appendix III: Comments from the Department of Defense: Appendix IV: GAO Contact and Staff Acknowledgments: Related GAO Products: Tables: Table 1: Changes in JSF Program Cost, Quantity, and Delivery Estimates: Table 2: Estimated Cost and Schedule for System Development: Table 3: Projected Procurement Funding Requirements for Fiscal Years 2010 through 2015: Table 4: Proposed Accelerated Procurement: Table 5: Overlap of Procurement Investments and Flight Testing: Table 6: Prime Contractor Schedule Management System Compared to Best Practices: Figures: Figure 1: Cumulative Procurement Costs and Quantities: Figure 2: JSF Labor Hour Estimates for Development Test Aircraft: Figure 3: Budgeted Manufacturing Hours by Development Aircraft (Wing and Mate and Delivery Stages): Figure 4: Comparison of Budgeted Labor Hours to Build Last Development Aircraft: Figure 5: JSF Planned Development Test Flights: Figure 6: Breakdown of Verification Venues for the JSF: Abbreviations: CAIG: Cost Analysis Improvement Group: CTOL: conventional takeoff and landing: DOD: Department of Defense: JSF: Joint Strike Fighter: MRP: Manufacturing Resource Planning: OSD: Office of the Secretary of Defense: STOVL: short takeoff and vertical landing: [End of section] United States Government Accountability Office: Washington, DC 20548: March 12, 2009: Congressional Committees: The Joint Strike Fighter (JSF) is the Department of Defense's (DOD) most complex and ambitious aircraft acquisition, seeking to simultaneously produce and field three aircraft variants for the Air Force, Navy, Marine Corps, and eight international partners. The JSF is critical to our nation's plans for recapitalizing the tactical air forces and will require a long-term commitment to very large annual funding outlays. The total expected investment is now more than $1 trillion--more than $300 billion to acquire 2,456 aircraft and $760 billion in life cycle operation and support costs, according to official program estimates. The Ronald W. Reagan National Defense Authorization Act for Fiscal Year 2005 requires GAO to review the JSF program annually for 5 years.[Footnote 1] Previous reports identified opportunities for the program to reduce risks and improve chances for more successful outcomes. We have expressed concern about the substantial overlap of development, test, and production activities and recommended a more evolutionary and knowledge-based acquisition strategy with limited investment in production aircraft until each variant demonstrates required capabilities in flight testing. Our most recent report questioned DOD's decision to reduce test aircraft and flight hours and recommended that a new comprehensive cost estimate be prepared.[Footnote 2] The department has not implemented our recommendations and cost and schedule increases have been the result. This is the fifth and final report under the mandate in which we (1) determine the JSF program's progress in meeting cost, schedule, and performance goals; (2) assess plans and risks in manufacturing and capacity to accelerate production; and (3) evaluate plans, progress, and risks with testing plans and risks in testing activities. To conduct this work, we tracked and compared current cost and schedule estimates with those of prior years, identified changes, and determined causes. We obtained program status reports, manufacturing data, and test planning documents. We assessed the program's scheduling estimates against best practices. We discussed results to date and future plans to complete JSF development and accelerate procurement with DOD, JSF, and contractor officials. Some of the cost data used in our report are based on cost projections that were current at the time of our review rather than the official program of record. Appendix I includes additional information about our scope and methodology. We conducted this performance audit from June 2008 to March 2009 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Results in Brief: JSF development will cost more and take longer than reported to the Congress last year, and DOD wants to accelerate procurement believing that will more quickly recapitalize tactical air forces. The program office estimates that an additional $2.4 billion is needed for cost overruns on the air system and engine contracts and for a 1-year extension to the development schedule. Its estimate does not include funding for the alternate engine program. An independent joint DOD cost estimating team identified a need for as much as $7.4 billion in additional funding for development through fiscal year 2015 and a 3- year schedule extension. This would increase total system development costs to $51.8 billion---a 17 percent increase from the April 2008 estimate--and delay completion of development to October 2016. The joint team's cost estimate was higher than the program office's estimate because it included the alternate engine effort directed by the Congress and made more conservative assumptions about engineer staffing levels, software requirements growth, manufacturing labor hours, and flight testing. Despite development cost increases and schedule delays, DOD officials want to accelerate JSF procurement by purchasing an additional 169 aircraft from fiscal years 2010 through 2015. This would require up to $33.4 billion in additional procurement funding for those 6 years and expose the government to additional risk from future cost increases because of the contract type. The plan would not increase the total JSF quantity through completion, but would buy these aircraft earlier than planned. DOD did not estimate the net effect this plan would have on future procurement funding to complete JSF acquisition. Ongoing manufacturing inefficiencies and parts problems have significantly delayed the delivery of needed test aircraft, and the prime contractor has not yet achieved the levels of learning expected, even as the program ramps up production. The contractor has extended manufacturing schedules three times and produced 2 of 13 development test aircraft. In the past year, budgeted hours to complete work on all test aircraft have increased by 40 percent. The program is still recovering from earlier problems in development--late release of engineering drawings resulting in design changes and delays in establishing the supplier base, late part deliveries, and inefficient manufacturing line work-arounds where unfinished work is completed out of station. The prime contractor has taken steps to improve manufacturing, the supplier base, and schedule management. A thorough schedule risk analysis could further enhance management insight into areas of uncertainty and better inform subsequent actions to correct persistent problems, such as schedule slippage and allocation of management reserves. Officials expect to deliver all test aircraft and address most of these problems by 2010. By that time, DOD plans to have procured 62 operational aircraft and will be quickly ramping up production. As DOD has a large backlog of production jets on order and is still working to deliver test jets and mature manufacturing processes, plans to accelerate procurement will be difficult to achieve in a cost-effective manner. DOD's revised test plan adds an extra year to the schedule, better aligns resources and availability dates, and lessens the overlap between development and operational testing, but it is still aggressive and allows little time for error discovery, rework, and recovery from downtime. DOD's decision late in 2007 to reduce the number of development test aircraft and to decrease the number of flight tests added risk, while any additional delays in delivering test aircraft will further compress the schedule. The independent cost team believes flight testing will require an additional 2 years to complete and suggests more flight hours to test mission systems and the carrier variant in particular. The revised plan relies on advanced and robust simulation labs, a flying test bed, and analytical studies to verify nearly 83 percent of the aircraft's capabilities while only 17 percent is to be verified through flight testing. While the labs appear more prolific, integrated, and capable than those used in legacy programs, the ability to substitute for flight testing has not yet been demonstrated. Significant overlap of development, test, and production schedules results in DOD making substantial investments before flight testing proves that the JSF will perform as expected. Under the accelerated procurement plan, DOD may procure 360 aircraft costing an estimated $57 billion before completing development flight testing. Acquiring large numbers of aircraft before testing successfully demonstrates that the design is mature, meets performance requirements, and is suitable could result in substantial future cost growth to correct deficiencies found during testing. To enhance congressional oversight and increase the likelihood of more successful program outcomes, we are recommending that the Under Secretary of Defense for Acquisition, Technology and Logistics report on the JSF's contracting strategy to the congressional defense committees by October 1, 2009. This report would include (1) an explanation of the remaining program risks and the factors justifying the continued use of cost reimbursement contracts for JSF's future low- rate production quantities, (2) the program's strategy for managing and mitigating risks associated with the use of cost contracts versus fixed- price contracts, and (3) plans for transitioning to fixed-price contracts for production, including time frames and criteria. To further maintain confidence that the program is on track to meet planned cost, schedule, and performance goals, we are also recommending that the JSF Program Office ensure that the prime contractor performs periodic schedule risk analyses for the JSF program to provide better insight into management reserve, production efficiencies, and schedule completion. Background: The JSF is a joint, multinational acquisition program for the Air Force, Navy, and Marine Corps and eight international partners. The program began in November 1996 with a 5-year competition between Lockheed Martin and Boeing to determine the most capable and affordable preliminary aircraft design. Lockheed Martin won the competition, and the program entered system development and demonstration in October 2001. Program goals are to develop and field an affordable, highly common family of stealthy, next-generation strike fighter aircraft for the Navy, Air Force, Marine Corps, and U.S. allies. The JSF is a single-seat, single-engine aircraft, designed to rapidly transition between air-to-ground and air-to-air missions while still airborne. To achieve its mission, the JSF will incorporate low- observable (stealth) technologies, defensive avionics, advanced onboard and offboard sensor fusion, internal and external weapons, and advanced prognostic maintenance capability. The JSF family consists of three variants. The conventional takeoff and landing (CTOL) variant will primarily be an air-to-ground replacement for the Air Force's F-16 Falcon and the A-10 Warthog aircraft, and will complement the F-22A Raptor. The short takeoff and vertical landing (STOVL) variant will be a multi-role strike fighter to replace the Marine Corps' F/A-18C/D Hornet and AV-8B Harrier aircraft. The carrier-suitable variant will provide the Navy a multi-role, stealthy strike aircraft to complement the F/A-18E/F Super Hornet. DOD is planning to buy a total of 2,456 JSFs and allies are expected to procure a minimum of 730 CTOL and STOVL aircraft. Because of the program's sheer size and the numbers of aircraft it will replace, the JSF is the linchpin of DOD's long-term plan to modernize tactical air forces. It is DOD's largest acquisition program, with total cost currently estimated at $300 billion, and the longest in planned duration, with procurement projected through 2034. In addition, the JSF remains DOD's largest cooperative program.[Footnote 3] Our international partners are providing about $4.8 billion toward development, and foreign firms are part of the industrial base producing aircraft. DOD's funding requirements for the JSF assume economic benefits from these foreign purchases in reducing unit costs for U.S aircraft. Table 1 shows the evolution of DOD's official estimated cost, quantity, and deliveries from the initiation of system development in October 2001 to the current official program of record dated December 2007 and submitted to the Congress in April 2008. It depicts quantities reduced in the last major program restructure in 2004, the impacts of increased costs on unit prices, and the slip in delivering initial operating capability to the warfighter. Table 1: Changes in JSF Program Cost, Quantity, and Delivery Estimates: Expected quantities: Development quantities; October 2001 (system development start): 14; December 2003 (2004 replan): 14; December 2006: 15; December 2007: 13. Expected quantities: Procurement quantities (United States only); October 2001 (system development start): 2,852; December 2003 (2004 replan): 2,443; December 2006: 2,443; December 2007: 2,443. Total quantities; October 2001 (system development start): 2,866; December 2003 (2004 replan): 2,457; December 2006: 2,458; December 2007: 2,456. Cost estimates (then-year dollars in billions): Development; October 2001 (system development start): $34.4; December 2003 (2004 replan): $44.8; December 2006: $44.2; December 2007: $44.4. Cost estimates (then-year dollars in billions): Procurement; October 2001 (system development start): 196.6; December 2003 (2004 replan): 199.8; December 2006: 255.1; December 2007: 254.0. Cost estimates (then-year dollars in billions): Military construction; October 2001 (system development start): 2.0; December 2003 (2004 replan): 0.2; December 2006: 0.5; December 2007: 0.5. Total program acquisition; October 2001 (system development start): $233.0; December 2003 (2004 replan): $244.8; December 2006: $299.8; December 2007: $298.9. Unit cost estimates (then-year dollars in millions): Program acquisition; October 2001 (system development start): $81; December 2003 (2004 replan): $100; December 2006: $122; December 2007: $122. Unit cost estimates (then-year dollars in millions): Average procurement; October 2001 (system development start): 69; December 2003 (2004 replan): 82; December 2006: 104; December 2007: 104. Estimated delivery dates: First operational aircraft delivery; October 2001 (system development start): 2008; December 2003 (2004 replan): 2009; December 2006: 2010; December 2007: 2010. Estimated delivery dates: Initial operational capability; October 2001 (system development start): 2010-2012; December 2003 (2004 replan): 2012-2013; December 2006: 2012-2015; December 2007: 2012-2015. Source: GAO analysis of DOD data. Notes: Data are from the annual selected acquisition reports that are dated in December of each year but are not officially released and reported to the Congress until March or April of the following year. The December 2003 data reflect the last major restructuring of the program. The December 2007 data represent the official program of record at the time of our review and was reported to the Congress in April 2008. Military construction costs have not been fully established, and the reporting basis changed over time in these DOD reports. [End of table] In our March 2008 report,[Footnote 4] we stated that JSF costs would likely grow much higher than reported because the program of record at that time did not include all acquisition costs (including the alternate engine program directed by the Congress), made overoptimistic assumptions, and did not fully reflect the mounting cost and schedule pressures from manufacturing inefficiencies and compressed time frames for completing development. We questioned the Mid-Course Risk Reduction Plan adopted by DOD in September 2007 that cut two development test aircraft, reduced test flights, and accelerated the reduction in the prime contractor's development workforce in order to replenish management reserves depleted by design changes and manufacturing problems. We recommended that DOD accomplish a full and comprehensive estimate of the total program verified by an independent third party and revisit the Mid-Course Risk Reduction Plan with an intensive analysis of the causes of management reserve depletion, progress against the baseline manufacturing schedule, and correction of deficiencies in the contractor's earned value management system. DOD agreed to make a comprehensive independent cost estimate, but decided to go ahead as planned with the Mid-Course Risk Reduction Plan, stating that it would monitor and evaluate progress and revise the plan later if it failed to achieve expectations. More Money and Time Will Be Needed to Complete JSF Development, While DOD Plans to Accelerate Procurement: Two recent estimates indicate that JSF development will cost more and take longer to complete than reported to the Congress in April 2008, primarily because of contract cost overruns and extended time to complete flight testing. DOD also plans to accelerate aircraft procurement over the next 6 years--buying more aircraft sooner than planned last year. This new plan will require significantly more procurement funding sooner, but officials did not assess its net effect on total program costs through completion of JSF acquisition. New Estimates Project Rising Costs and Further Delays to Complete JSF Development: Development costs are projected to increase between $2.4 billion and $7.4 billion and the schedule for completing system development extended from 1 to 3 years, according to recent estimates--one by the JSF Program Office and one by a joint team of Office of the Secretary of Defense (OSD), Air Force, and Navy officials.[Footnote 5] Cost overruns on both the aircraft and engine contracts, delays in manufacturing test aircraft, and a need for a longer, more robust flight test program are the primary cost drivers. The joint team's estimate is higher than the program office's because it included costs for the alternate engine program directed by the Congress and used more conservative assumptions based on current and legacy aircraft experiences. Program officials contend that funding the program to the higher cost estimate is premature and believe processes are in place to substantially improve on the test experiences of past programs. Regardless, both estimates agree that cost and time to complete development have increased from the official program of record at the time of our review. (See table 2.) Table 2: Estimated Cost and Schedule for System Development: Development costs to complete; 2007 program of record: $7.4 billion; JSF Program Office: $9.8 billion; Joint estimating team: $14.8 billion[A]. Total development costs; 2007 program of record: $44.4 billion; JSF Program Office: $46.8 billion; Joint estimating team: $51.8 billion[A]. Date to complete development; 2007 program of record: October 2013; JSF Program Office: October 2014; Joint estimating team: October 2016. Source: GAO analysis of DOD data. [A] The joint estimating team only projected costs through fiscal year 2015. Extending development to October 2016, as the team projects, would increase both cost figures. DOD data suggest that 1 year of additional flight testing and other government costs could be about $700 million. [End of table] The program office's revised development cost estimate projects an additional $2.4 billion and a 1-year extension in the schedule compared to the official program of record reported to the Congress. This would increase the system development portion of the acquisition program to $46.8 billion and delay its completion to October 2014. The cost increases primarily resulted from the following factors. * $1.2 billion for aircraft development. Program officials declared a cost overrun on the prime air system contract because of increased labor hours, higher prices, and supply shortages. Included in this figure is $200 million to be added to the contractor's management reserve.[Footnote 6] Last year, we reported[Footnote 7] that mounting cost and schedule pressures depleted reserves much faster than anticipated. By summer 2007, the program had spent about two-thirds of budgeted funds but had accomplished only half the work required. Since then, DOD's efforts to restore reserves and fix manufacturing inefficiencies have not fully achieved intended results, requiring another cash infusion. * $800 million for engine development. According to officials from the Defense Contract Management Agency, the engine contractor continued to face development problems, which resulted in a contract cost overrun. Higher costs for labor and materials, supplier problems, and the rework needed to correct deficiencies with the engine blade design discovered during ground testing were major contributing factors. * $300 million for flight test extension. The program extended system development by 1 year to provide more time for development and operational flight testing. In April 2008, an operational test review team recommended extending the development contract by 1 year. The review team considered but dismissed several other options to address the schedule problem, including deferring requirements. On the other hand, the joint estimating team estimates that it will cost $14.8 billion to complete JSF development, $7.4 billion more than the official program of record at the time of our review. This would increase the total development costs to $51.8 billion from the $44.4 billion reported to the Congress last year--a 17 percent increase. The joint team also projected a 3-year program extension beyond the program of record in order to complete system development, 2 years more than the new program office estimate. The joint team's estimate was $5 billion more than the new program office estimate.[Footnote 8] Several factors account for the difference between the two estimates. * Alternate engine. The joint estimating team included $1.4 billion to complete development of an alternate engine for the JSF; the program office estimate did not include alternate engine costs. The Congress has directed DOD to develop a second source for the JSF engine to induce competition and to ensure that one engine's failures would not ground all JSFs, thereby reducing operational risks in the future. DOD has not wanted to pursue this second engine source and twice removed funding from the JSF program line. * Engineering staffing. The joint estimating team projected a need for the contractor to retain considerably more engineering staff and for longer periods of time than the program office estimate to complete development, evaluate test results, and correct deficiencies. Releasing engineering staff prematurely risks not discovering problems until late in development or during fielding, when they would be more expensive to address. * Software development. The joint estimating team believes that the software productivity rate will be less than the program's calculation and anticipates much more growth in software requirements. The JSF aircraft is expected to require 7.5 million lines of computer code--the most by far of any aircraft. By comparison, the F/A-18E/F has only 1.1 million and the F-22A 2.2 million. Experiences on past acquisitions have shown 30 to 100 percent growth in software requirements over time, while the JSF Program Office estimate assumed no growth. * Flight testing. The joint estimating team projects that flight testing will require more time and effort than the program office has built into its current schedule. Continuing delays in delivering test aircraft are expected to hamper and further compress test schedules. In particular, the joint team projects that the two aircraft dedicated to carrier suitability tests will be late off the production line, thereby delaying test activities. It also projected that the JSF will require about 2,700 hours of flight testing for mission systems, significantly more than the 1,700 hours that the program office currently estimates. * Manufacturing production hours. The joint estimating team projects that production span times for the JSF will be longer than the program office estimates based on the program's performance to date and experience of recent programs, such as the F/A-18 E/F and the F-22A. The span time is an indicator of how long the manufacturing effort takes and when flight testing can begin. The program office assumes that span times will decrease over the course of the development contract. We note that span times typically increase during development, as was the case for both the B-2 bomber and F-22A programs. Program officials believe that their estimate is more accurate and that providing extra funding to address future risks is premature and does not provide incentives for contractors to control costs. The program office attributes its lower cost estimate to several factors. First, the quantity, quality, and flexibility of the JSF laboratories should enable the program to reduce more risks in a laboratory environment, rather than through flight testing, which is considerably more expensive. In addition, the program's efforts to develop the final software system infrastructure early should reduce significant software problems later in the program, according to the program office. The program office also believes that costs will be lower because progress in several key development areas is either matching plans or is ahead of where legacy programs were at similar points of time in their development. For example, the program is currently reducing engineering staff as planned. The program is also producing software at a rate significantly higher than that of the F-22A program and is at least 18 months ahead of where the F-22A program was at a similar point in developing mission systems, according to program officials. Officials told us that they intend to fund the fiscal year 2010 development budget based on the joint team's higher estimate. However, it is not clear at this stage which estimate will serve as the basis for future budget submissions. Much Higher Annual Procurement Funding Required to Accelerate JSF Procurement: The program office and joint estimating team also projected procurement funding requirements for the 6-year period fiscal years 2010-2015 based on DOD plans to accelerate procurement of operational aircraft. Through this effort, DOD wants to recapitalize tactical air forces sooner and mitigate projected fighter shortfalls in the future. Compared to last year, this accelerated plan would procure an additional 169 aircraft during these 6 years, moving aircraft that had been scheduled for procurement beyond 2015 to earlier years. According to the two estimates, this plan would require from $21.8 billion to $33.4 billion more funding than the official program of record, as shown in table 3. Table 3: Projected Procurement Funding Requirements for Fiscal Years 2010 through 2015: Procurement funding requirements; 2007 program of record: $59.7 billion; JSF Program Office: $81.5 billion; Joint estimating team: $93.1 billion. Procurement quantity; 2007 program of record: 485; JSF Program Office: 654; Joint estimating team: 654. Average procurement unit cost; 2007 program of record: $123 million; JSF Program Office: $125 million; Joint estimating team: $142 million. Source: GAO analysis of DOD data. [End of table] The joint team's estimate is higher than the program office's, primarily for these reasons: * The joint team projected slower gains in production efficiency than the program office. Typically, production efficiency is improved and unit costs are lowered over time as a workforce becomes more experienced building a new product and manufacturing processes are honed. * The joint team also assumed fewer savings from commonality. Commonality--a key selling point for the JSF program--refers to the use of the same or similar parts, structures, and subsystems shared by the three variants. Greater commonality can save money by decreasing development times and facilitating economic order quantities. * The team projected higher labor and material costs and longer production span times, based on JSF performance to date in manufacturing development test aircraft. Table 4 shows the additional aircraft and funding requirements for DOD's accelerated plan compared to the official program of record. These quantities are for the United States only; during this same period, the international partners are expected to buy 273 aircraft. Table 4: Proposed Accelerated Procurement: 2007 program of record funding requirements[A]; FY10: $6.0; FY11: $6.9; FY12: $10.5; FY13: $10.8; FY14: $12.4; FY15: $13.1; Total: $59.7. 2007 program of record quantities; FY10: 30; FY11: 43; FY12: 82; FY13: 90; FY14: 110; FY15: 130; Total: 485. Program office accelerated funding requirements[A]; FY10: $8.0; FY11: $12.8; FY12: $14.3; FY13: $14.4; FY14: $15.3; FY15: $16.7; Total: $81.5. Joint team accelerated funding requirements[A]; FY10: $8.9; FY11: $13.9; FY12: $16.7; FY13: $16.6; FY14: $17.8; FY15: $19.3; Total: $93.1. Accelerated quantities; FY10: 32; FY11: 70; FY12: 109; FY13: 119; FY14: 146; FY15: 178; Total: 654. Source: GAO analysis of DOD data. [A] Funding requirements are expressed in billions of then-year dollars. [End of table] Procurement of Operational Aircraft under Cost Reimbursement Contracts to Continue; Increases the Government's Exposure to Risks: The JSF program is procuring a substantial number of aircraft on cost reimbursement contracts. Cost contracts place most of the risk on the buyer--DOD in this case--who is liable to pay more than budgeted should labor, material, or other incurred costs be more than expected when the contract was signed. JSF officials plan to procure at least the first four low-rate production lots under cost reimbursement contracts and to transition to fixed-price instruments when appropriate, possibly between lots 5 and 7 (fiscal years 2011 to 2013). It is unclear exactly how and when this will happen, but the expectation is to transition to fixed pricing once the air vehicle has a mature design, has been demonstrated in flight tests, and is producible at established cost targets. To date, DOD has procured the first three lots for a total of 30 aircraft and $7.4 billion on cost reimbursement terms. Under the accelerated procurement plan, DOD could procure as many as 360 aircraft costing about $57 billion through fiscal year 2013 on cost reimbursement contracts, as illustrated in figure 1. Figure 1: Cumulative Procurement Costs and Quantities: [Refer to PDF for image] Cumulative Cost: Line: FY06: 0.1; FY07: 0.9; FY08: 3.6; FY09: 7.4; FY10: 15.4; FY11: 28.2; FY12: 42.5; FY13: 56.9; FY14: 72.2; FY15: 88.9. Bar: Between FY11 and FY 13: Period of possible transition from cost contracts to fixed-price contracts. Cumulative Quantity: Line: FY06: 0; FY07: 2; FY08: 14; FY09: 30; FY10: 62; FY11: 132; FY12: 241; FY13: 360; FY14: 506; FY15: 684. Bar: Between FY11 and FY13: Period of possible transition from cost contracts to fixed-price contracts. Source: GAO analysis of DOD data. [End of figure] Cost reimbursement contracts provide for payment of allowable incurred costs, to the extent prescribed in the contract. According to the Federal Acquisition Regulation, cost reimbursement contracts are suitable for use only when uncertainties involved in contract performance do not permit costs to be estimated with sufficient accuracy to use any type of fixed-price contract.[Footnote 9] Cost reimbursement contracts for weapon production are considered appropriate when the program lacks sufficient knowledge about system design, manufacturing processes, and testing results to establish firm prices and delivery dates. In contrast, a fixed-price contract provides for a pre-established price, places more of the risk and responsibility for costs on the contractor, and provides more incentive for efficient and economical performance. Procuring up to 360 production aircraft on cost reimbursement contracts---nearly 15 percent of the total DOD program--seems to be a tacit acknowledgment by DOD and the contractor that knowledge on JSF design, production processes, and costs for labor and material is not yet sufficiently mature and that pricing information is not exact enough for the contractor to assume the risk under a fixed-price contract. It also seems to be a consequence of the substantial concurrency of development, test, and production built into the JSF schedule. Significant overlap of these activities means that DOD is procuring considerable quantities of operational aircraft while development test aircraft are still on the production line and much in advance of testing to prove aircraft performance and suitability. Establishing a clear and accountable path to ensure that the contractor assumes more of the risk is prudent. We note that the significant ramp up in JSF production under the accelerated profile starts with lot 5, the fiscal year 2011 procurement of 70 aircraft. Continued Manufacturing Inefficiencies Will Make It Difficult for the Program to Meet Its Production Schedule: Manufacturing of development test aircraft is taking more time, money, and effort than planned, but officials believe that they can work through these problems and deliver the 11 remaining test aircraft by early 2010. However, by that time, DOD may have procured as many as 62 production aircraft, accumulating a backlog of aircraft to be produced. Manufacturing inefficiencies and parts shortages continue to delay the completion and delivery of development test aircraft needed for flight testing. The contractor has not yet demonstrated mature manufacturing processes, or an ability to produce at currently planned annual rates. It has taken steps to improve manufacturing, the supplier base, and schedule management. However, given the manufacturing challenges, we believe that DOD's plan to accelerate production in the near term adds considerable risk and will be difficult to achieve in a cost-effective manner. Time and Money Needed for Manufacturing Development Test Aircraft Continue to Increase: The prime contractor has restructured the JSF manufacturing schedule three times, each time lengthening the time to deliver aircraft to the test program. Delays and manufacturing inefficiencies are prime causes of contract cost overruns. The contractor has produced two development test aircraft--an original non-production representative model and the first STOVL aircraft. It now projects delivering the remaining 11 aircraft in 2009 and early 2010. Problems and delays are largely the residual effects from difficulties early in development. The effects of the late release of engineering drawings, design changes, delays in establishing a supplier base, and parts shortages continue to cause delays and force inefficient production line work-arounds where unfinished work is completed out of station.[Footnote 10] Data provided by the Defense Contract Management Agency and the JSF Program Office show continuing critical parts shortages, out-of-station work, and quality issues. The total projected labor hours to manufacture test aircraft increased by 40 percent just in the past year, as illustrated in figure 2. Figure 2: JSF Labor Hour Estimates for Development Test Aircraft: [Refer to PDF for image] Aircraft: BF-1; Budgeted hours (June 2007): 343; Budgeted hours (June 2008): 358. Aircraft: BF-2; Budgeted hours (June 2007): 233; Budgeted hours (June 2008): 294. Aircraft: BF-3; Budgeted hours (June 2007): 180; Budgeted hours (June 2008): 251. Aircraft: BF-4; Budgeted hours (June 2007): 155; Budgeted hours (June 2008): 229. Aircraft: AF-1; Budgeted hours (June 2007): 188; Budgeted hours (June 2008): 258. Aircraft: AF-2; Budgeted hours (June 2007): 146; Budgeted hours (June 2008): 218. Aircraft: AF-3; Budgeted hours (June 2007): 131; Budgeted hours (June 2008): 204. Aircraft: CF-1; Budgeted hours (June 2007): 239; Budgeted hours (June 2008): 255. Aircraft: CF-2; Budgeted hours (June 2007): 126; Budgeted hours (June 2008): 210. Aircraft: CF-3; Budgeted hours (June 2007): 118; Budgeted hours (June 2008): 196. Aircraft: BF-5; Budgeted hours (June 2007): 103; Budgeted hours (June 2008): 174. Aircraft: AF-4; Budgeted hours (June 2007): 91; Budgeted hours (June 2008): 165. Source: GAO analysis of DOD data. [End of figure] Performance data for two major cost areas--wing assembly and mate and delivery--indicate even more substantial growth. Figure 3 compares the increased budgeted hours in the new schedule to last year's estimates. The 2007 schedule assumed a steeper drop in labor hours as more units are produced and manufacturing and worker knowledge increases; this increased efficiency because of learning is typical of production programs. The new schedule, based upon actual performance, projects a less steep decline in labor hours, indicating lesser gains in worker efficiency. As of June 2008, the planned hours for these two major stations increased by about 90 percent over the June 2007 schedule, which itself had shown an increase from the 2006 schedule. Figure 3: Budgeted Manufacturing Hours by Development Aircraft (Wing and Mate and Delivery Stages): [Refer to PDF for image] Wing manufacturing data: Budget labor hours (in thousands): Aircraft: BF-1; June 2007: 70,612; June 2008: 76,205. Aircraft: BF-2; June 2007: 55,280; June 2008: 90,555. Aircraft: BF-3; June 2007: 42,892; June 2008: 75,345. Aircraft: BF-4; June 2007: 40,205; June 2008: 71,645. Aircraft: AF-1; June 2007: 48,925; June 2008: 76,577. Aircraft: AF-2; June 2007: 39,735; June 2008: 72,038. Aircraft: AF-3; June 2007: 36,711; June 2008: 66,727. Aircraft: CF-1; June 2007: 49,287; June 2008: 90002. Aircraft: CF-2; June 2007: 28,605; June 2008: 80,611. Aircraft: CF-3; June 2007: 23,694; June 2008: 71,858. Aircraft: BF-5; June 2007: 16,617 June 2008: 63,734. Aircraft: AF-4; June 2007: 17,084; June 2008: 52,820. Mate and delivery manufacturing data: Budget labor hours (in thousands): Aircraft: BF-1; June 2007: 69683; June 2008: 71113. Aircraft: BF-2; June 2007: 52372; June 2008: 67737. Aircraft: BF-3; June 2007: 35245; June 2008: 55642. Aircraft: BF-4; June 2007: 26260; June 2008: 56623. Aircraft: AF-1; June 2007: 24493; June 2008: 51046. Aircraft: AF-2; June 2007: 18210; June 2008: 45924. Aircraft: AF-3; June 2007: 16972; June 2008: 51728. Aircraft: CF-1; June 2007: 18419; June 2008: 47558. Aircraft: CF-2; June 2007: 16275; June 2008: 43657. Aircraft: CF-3; June 2007: 20900; June 2008: 49591. Aircraft: BF-5; June 2007: 14214; June 2008: 39296. Aircraft: AF-4; June 2007: 9735; June 2008: 43980. Source: GAO analysis of DOD data. [End of figure] The overlap in the work schedule between manufacturing the wing and mating (connecting) it to the aircraft fuselage has been a major concern for several years because it causes inefficient out-of-station work. The contractor continues to address this concern, but the new schedule indicates that this problem will continue at least through 2009. One indicator of its persistence is the projected hours for building the last test aircraft. As figure 4 shows, estimated labor hours increased more than 80 percent from the June 2007 to June 2008 schedules, and the planned hours for wing assembly and for the mate and delivery phases more than tripled. Figure 4: Comparison of Budgeted Labor Hours to Build Last Development Aircraft: [Refer to PDF for image] Pie graphs: June 2007: Male and delivery: 11%: 9,735; Wing: 19%: 17,084; Other components: 70%: 64,399; 91,218 budgeted hours. June 2008: Male and delivery: 27%: 43,980; Wing: 32%: 52,820; Other components: 41%: 68,640; 165, 440 budgeted hours. Source: GAO analysis of DOD data. [End of figure] Prime Contractor Actions to Improve Schedule Management, Manufacturing Efficiency, and Supplier Base: Our evaluation determined that the prime contractor now has good tools and integrated processes in place that should improve its schedule management activities and is also implementing actions to improve manufacturing efficiency, the delivery of parts, and proactive oversight of subcontractors. The effects from these recent actions are not yet fully apparent, and the contractor has not yet accomplished its own schedule risk assessment that could provide more insight into impacts from areas of risk and uncertainty. The coming year will be critical for implementing management improvements in order to accomplish a firm and effective transition from manufacturing a few test aircraft to producing operational aircraft at high annual rates. Prime Contractor's Scheduling Management Processes Meet Many Best Practices, but Program Risks Are Not Entirely Visible: The prime contractor demonstrated to us that its schedule management processes largely meet established best practices criteria. With improvements implemented in 2008, the contractor's management systems meet or partially meet eight of the nine established criteria.[Footnote 11] For example, the master schedule can identify and track activities associated with over 600 projects. It also establishes the critical path between activities, allowing the program to examine the impacts of schedule delays and determine schedule flexibility. Appendix II discusses our examination of the prime contractor's schedule management process against best practices criteria in more detail. The one area not meeting best practices was related to performing a schedule risk analysis, and as a result, the contractor has limited insight into areas of risk and uncertainty in the schedule. The prime contractor has not conducted its own risk assessment that would (1) determine the level of confidence it has in meeting completion dates and (2) identify and apportion reserve funds for contingencies. A thorough risk analysis could improve management insight and subsequent corrective actions on two recurring problem areas in particular: schedule slippage and inadequate management reserve levels. Naval Air Systems Command officials did accomplish an independent schedule risk analysis that indicated that the program could slip more than 2 years based on the productivity risks associated with software development and assembly of the various airframes as well as the time needed to complete all flight testing. Both the contractor and the JSF Program Office disputed the findings of the Naval Air Systems Command schedule risk analysis primarily because the analysis was done without direct involvement of program officials. Improvements in Manufacturing and Supplier Base Are Ongoing: The prime contractor is implementing changes designed to address the manufacturing inefficiencies and parts shortages discussed earlier. These include (1) increasing oversight of key subcontractors that are having problems, (2) securing long-term raw material purchase price agreements for both the prime and key subcontractors, and (3) implementing better manufacturing line processes. On this latter point, according to program officials, the prime contractor has taken specific steps to improve wing manufacturing performance--one of the most troublesome workstations. Defense Contract Management Agency officials noted that the contractor produced the second STOVL aircraft variant with less work performed out of station than for the first STOVL aircraft. Also, program office and contractor officials report some alleviation of parts shortages and improvements in quality, but also believe that the effects from previous design delays, parts shortages, and labor inefficiencies will continue to persist over the near term. The lag time between taking action and demonstrating improvement may partly explain why some manufacturing performance metrics are not demonstrating a clear continued rate of quality improvement, as would be desirable and expected for a program ramping up annual production rates. This lag time may be evident in two important metrics--scrap, rework, and repair rates and manufacturing defect rates--both of which have increased somewhat since 2006. Program and contracting officials point out, however, that while this performance is not desirable, these and other metrics compare very favorably with those of prior acquisitions at similar stages of development, including the F-16 and F- 22A. Supplier costs are expected to make up an even more substantial share of total expenses as the program moves further into production. According to contractor officials, efforts are focused on maturing the supply base and working more closely and directly with key suppliers to reduce costs, alleviate parts shortages, and support higher production rates. Key suppliers have struggled to develop critical and complex parts while others have had problems with limited production capacity. For example, the supplier responsible for the advanced electro- hydraulic actuation system had delivered parts with missing subcomponents and parts that were not built to specifications. The major "team mate" supplier responsible for fuselage and tail assembly has experienced delays caused by its limited machining capacity. Given these supplier issues, manufacturing inefficiencies, and accumulating backlog in production, we believe that the program's plans to accelerate production in the near term adds considerable risk and will be difficult to achieve in a cost-effective manner. The JSF's Test Plan Is Improved but Faces Numerous Challenges to Complete Development on Time and on Budget: DOD will make significant investments--in both dollars and the number of aircraft procured--before completing JSF flight testing. DOD's proposal to accelerate procurement further increases financial risks in a very challenging test environment. DOD's new test plan adds an extra year to the schedule and better aligns resources, but is still aggressive with little room for error, presenting a formidable challenge to complete system development, support initial operational testing, and, eventually, a full rate production decision. DOD decisions to reduce development test aircraft and flight tests add to the risks, while any additional delays in manufacturing test aircraft will further compress the schedule. The JSF has just begun development flight testing with two test aircraft and has already experienced some setbacks--normal in any program, but of special concern when assets are minimal. Some in DOD forecast that another 2 or more years beyond the 1- year extension just approved will eventually be needed to successfully prove aircraft performance and complete system development. The department has stated that the contractor's state-of- the-art ground test labs and a flying test bed will mitigate risks in the flight regimen and their use will effectively substitute for flight testing. This approach is promising, but not yet proven. Significant Investments Made before Development Flight Tests Are Completed: DOD is investing heavily in procuring JSF aircraft before flight testing proves that they will perform as expected. Procuring aircraft before testing successfully demonstrates that the design is mature and that the weapon system will work as intended increases the likelihood of design and requirements changes resulting in subsequent cost growth, schedule delays, and performance limitations. Also, systems already built and fielded may later require substantial modifications, further adding to costs. The uncertain environment as testing progresses is one reason why the prime contractor and DOD are using cost reimbursable contracts until rather late in procurement. Table 5 depicts planned investments--in both dollars and aircraft--prior to the completion of development flight testing. Under the accelerated production plan and using the lower procurement cost estimate prepared by the program office, DOD may procure 360 aircraft at a total estimated cost of $57 billion before development flight testing is completed. This overlap will be further exacerbated should the joint estimating team's predictions of higher cost and lengthier schedule prove accurate. Table 5: Overlap of Procurement Investments and Flight Testing: Cumulative procurement (billions of then-year dollars); 2007: $0.9; 2008: $3.6; 2009: $7.4; 2010: $15.4; 2011: $28.2; 2012: $42.6; 2013: $57.0; 2014: $72.3. Cumulative quantity; 2007: 2; 2008: 14; 2009: 30; 2010: 62; 2011: 132; 2012: 241; 2013: 360; 2014: 506. Contract type; 2007: Cost; 2008: Cost; 2009: Cost; 2010: Cost; 2011: Cost or fixed; 2012: Cost or fixed; 2013: Cost or fixed; 2014: Fixed. Percentage of flight test program completed; 2007: <1%; 2008: <1%; 2009: 2%; 2010: 9%; 2011: 34%; 2012: 62%; 2013: 88%; 2014: 100%. Limited Knowledge Gained From Flight Testing-More Knowledge Gained From Flight Testing. Source: GAO analysis of DOD data. Notes: Flight testing data reflect the percentage of total development flight tests completed at the time of the planned investment decision, which is expected in January of each year. [End of table] Flight Testing Is Still in Its Infancy and Has Fallen Behind Schedule: The JSF program had completed about 2 percent of its development flight testing as of November 2008. Figure 5 shows the expected ramp up in flight testing with most effort occurring in fiscal years 2010 through 2012. Past programs have shown that many problems are not discovered until flight testing. As such, the program is likely to experience considerable cost growth in the future as it steps up its flight testing, discovers new problems, and makes the necessary technical and design corrections. Figure 5: JSF Planned Development Test Flights: [Refer to PDF for image] Fiscal year: 2007; Test flights: 27. Fiscal year: 2008; Test flights: 97. Fiscal year: 2009; Test flights: 317. Fiscal year: 2010; Test flights: 1243. Fiscal year: 2011; Test flights: 1425. Fiscal year: 2012; 1289. Fiscal year: 2013; Test flights: 593. Source: GAO analysis of DOD data. [End of figure] While the program has been able to demonstrate basic aircraft flying capabilities, it has recently experienced testing delays and has fallen behind the flight test plan established in 2007. At the time of our review, the program had flown about half of its planned 155 flight tests for 2008. The test program currently has two development test aircraft and an integrated airborne test bed, with the following experiences to date: * Sixty-five test flights on the original, non-production representative prototype contributed to discoveries in, among other things, landing gear door fitting, aerial refueling operations, and weapons bay functions. The prototype experienced a 3-month delay because of engine bay nacelle vent fan malfunctions that were subsequently resolved. * Initial testing of the first of 12 production representative prototypes began in June 2008--a STOVL variant flown in conventional mode. By the time of our review it accumulated 14 test flights demonstrating important handling qualities and reducing risks associated with, among other things, the landing gear, fuel system performance, and STOVL doors operation. Engine problems delayed full STOVL testing by 6 months. * Thirty-seven flights on the cooperative avionics test bed tested mission system software and demonstrated communication and navigation capabilities. Looking ahead, the program expects to take delivery of the remaining development test aircraft during 2009 and early in fiscal year 2010. In the same time frame, it also plans to begin flight testing 6 of its 12 production representative prototype test aircraft (CTOL and STOVL aircraft), including the first 2 aircraft dedicated to mission system testing. The first carrier variant development test aircraft is expected to begin full flight testing--including ship suitability testing--in 2010. A fully integrated, mission capable aircraft is not expected to enter flight testing until 2012 by which time DOD plans to have already purchased 241 aircraft for about $43 billion under cost reimbursement contracts. Program's Test Plan Extends Development and Relies Heavily on Ground Testing and Simulations to Verify Aircraft Performance: The JSF Program Office developed a new and improved test plan in the spring of 2008 that extended the development period by 1 year, better aligned test resources and availability dates, and lessened the overlap between development and operational testing. The new plan is still aggressive, however, and has little room for error discovery, rework, and recovery from downtime should test assets be grounded or otherwise unavailable. The sheer complexity of the JSF program--with 7.5 million lines of software code, three variants, and multi-mission development- -suggests that the aircraft will encounter many unforeseen problems during flight testing requiring additional schedule to rework. Our past work has shown that programs that do not allow sufficient time to address the inevitable problems discovered in testing, run a greater risk of significant cost increases and schedule delays when problems do arise. The joint estimating team noted that the program's flight test plan assumed a higher productivity than has been seen on recent flight test programs. As such, the joint team believes that flight testing will require an additional 2 years to complete beyond the 1 year already added to development and suggests that more flight test hours will be necessary to test mission systems and the carrier variant in particular. The Mid-Course Risk Recovery Plan, approved in late 2007, cut two development test aircraft, reduced test flights, and relies more on ground laboratories and simulations to verify performance, adding substantial risk to the program. Our 2008 report[Footnote 12] discussed the objections from several prominent DOD offices to the Mid-Course Risk Recovery Plan. The Director of Operational Test and Evaluation, for example, identified risks in the revised verification strategy and cited inadequate capacity to handle the pace of mission testing and ship suitability and signature testing. This increases the likelihood of not finding and resolving critical design deficiencies until operational testing, when it is more costly and disruptive to do so. The test plan relies heavily on a series of advanced and robust simulation labs and a flying test bed to verify aircraft and subsystem performance. Figure 6 shows that 83 percent of the aircraft's capabilities are to be verified through labs, the flying test bed, and subject matter analysis, while only 17 percent of test points are to be verified through flight testing. The JSF program spent $5 billion on its system of simulation labs and models. Program officials argue that their heavy investment in simulation labs will allow early risk reduction, thereby reducing the need for additional flight testing, controlling costs, and meeting the key milestones of the program's aggressive test plan. Figure 6: Breakdown of Verification Venues for the JSF: [Refer to PDF for image] Pie graph: Labs: 59%; Subject matter analysis: 19%; Flight testing: 17%; Flying test bed: 5%. Source: GAO analysis of DOD data. [End of figure] The JSF program's simulation labs appear more prolific, integrated, and capable than the labs used in past aircraft programs. The program utilizes 18 labs for development whereas only 9 were used for the F- 22A, 7 for the F-18, and 5 for the F-16. According to program officials, the greater number of labs allows engineers to work simultaneously on different development blocks, reducing bottlenecks that may occur in testing. In contrast, engineers of the F-18 and F-22A programs had to interrupt or shut down development on one development block while they were making corrections to another. Also in contrast to past programs, key JSF simulation labs are colocated at a Lockheed Martin plant in Fort Worth, Texas. The F-22A program utilized three locations and two different companies. According to the program office, the colocation of the key testing labs facilitates more seamless integration of key aircraft components. Program officials also noted that JSF labs use the actual aircraft components to a greater extent than labs did in past programs and also have greater software processing capacity. This allows for more realistic data, which should reduce the need for additional flight testing. Further, the JSF utilizes the first fully integrated airborne test bed for mission system testing. According to program officials, the test bed's design is geospatially proportionate to an actual F-35 aircraft, enhancing its ability to accurately verify aircraft performance. While the labs appear more prolific, integrated, and complex than those used in legacy programs, concerns about their extensive use in verifying aircraft performance remain. The extent of the JSF program's planned lab use is unprecedented, but the ability to substitute for flight testing has not yet been demonstrated. In addition, the labs have yet to be fully accredited. Accreditation is required to ensure the adequate capability of labs and models to perform verification tasks. It is critical that the models behave like aircraft to ensure that the system's performance requirements are being verified accurately. The program office said that it is on track to complete the accreditation of the labs in time to begin verifying system performance. However, the Director of Operational Test and Evaluation reports that the progress of the accreditation support packages is behind schedule and suggests that more flight testing may be needed as the accreditation process reveals the limitations of the models. Some DOD officials are also concerned that the labs will be understaffed. The Director of Operational Test and Evaluation and DOD's joint estimating team both reported that the program's current resource plans reduce engineering staff too rapidly. Engineering and test personnel are critical to analyzing the data generated from the labs. Without adequate staff, there is a greater risk that the labs will not be sufficiently utilized, which could, in turn, result in schedule delays or cost increases. Early Operational Assessment Identifies Several Challenges That May Have Operational Impact If Not Addressed: While the program is projecting that it will meet all key performance parameters, most will not be verified through ground and flight testing until fiscal years 2010 through 2013. In addition, a 2008 operational assessment by the Air Force Operational Test and Evaluation Center pointed out several technical challenges that it believes are likely to have a severe operational impact if not adequately addressed. While some of the report's concerns are not specific requirements, some DOD officials believe that the shortfalls may adversely affect the JSF's ability to meet warfighter needs. For example: * The current F-35 power system may cause excessive damage to runway surfaces which could limit its ability to operate in certain locations. The program is still evaluating the problem and plans to gather data and conduct further studies when full-scale models or actual aircraft are available. According to a program official, changes to the aircraft's design or to the current concept of operations may be needed. The program has alerted the services and believes it will have a better understanding of the problem sometime in 2009. * Thermal management challenges hamper the ability to conduct missions in hot and cold environments. The Director of Operational Test and Evaluation reported that an alternative main engine fuel pump to remedy this problem is under development but will not be available before the low-rate initial production Lot 3, which is likely to affect operational testing. The test team aborted a test sortie because of high fuel temperatures in June. * The JSF's advanced integrated support system aims to improve and streamline aircraft logistics and maintenance functions in order to reduce life cycle costs. The current integrated support system for the JSF prohibits operating two detachments from one squadron simultaneously. This limitation will severely affect current operating practices, which include dividing one squadron of aircraft into subgroups to deploy and operate at different locations. Conclusions: The JSF is DOD's largest and most complex acquisition program and the linchpin of efforts to recapitalize our tactical air forces. It is now in its most challenging phase, at a crossroads of a sort. Challenges are many--continuing cost and schedule pressures; complex, extensive, and unproven software requirements; and a nascent, very aggressive test program with diminished flight test assets. Looking forward, the contractor plans to complete work expeditiously to deliver the test assets, significantly step up flight testing, begin verifying mission system capabilities, mature manufacturing processes, and quickly ramp up production of operational aircraft. As such, the credibility of the program's test plans, manufacturing capacity, and subsequent cost and schedule estimates should become more apparent. The program must move forward, but given all these challenges, accelerating procurement in a cost-reimbursement contract environment- -where uncertainties in contract performance do not permit costs to be estimated with sufficient accuracy to use any type of fixed-price contract--places very significant financial risk on the government. Accelerating plans also does not equate to an ability to deliver to those plans. Because the program's manufacturing processes are still maturing and flight testing is still in its infancy, incorporating an accelerated production schedule introduces even more risk and uncertainty to the program. Our past work has shown that programs that make production decisions prior to fully proving a system's design through testing and demonstration of mature manufacturing processes have an increased risk of design and production changes and retrofits of completed aircraft.[Footnote 13] Until the contractor demonstrates that it can produce aircraft in a timely and efficient manner, DOD cannot fully grasp future funding requirements. DOD needs tangible assurance from the prime contractor that it can meet expected development and production expectations. By accelerating low-rate production quantities before manufacturing and testing processes are mature, DOD accepts most of the contractors' production and manufacturing inefficiencies. At minimum, the contractor needs to develop a detailed plan demonstrating how it can successfully meet program development and production goals in the near future within cost and schedule parameters. With an improved contracting framework and a more reasoned look to the future, the JSF program can more effectively meet DOD and warfighter needs in a constrained budget environment. Recommendations for Executive Action: Given the program's ongoing manufacturing problems and nascent flight test program, we believe that moving forward with an accelerated procurement plan is very risky. This risk is reflected by the extended use of cost reimbursement contracts for low-rate production quantities- -a contract mechanism that places most of the cost risk on the government. As such, to enhance congressional oversight, increase the likelihood of more successful program outcomes, and maintain confidence that the program is on track to meet planned cost, schedule, and performance goals, we recommend that the Secretary of Defense take the following two actions: 1. Direct the Under Secretary of Defense for Acquisition, Technology and Logistics to report to the congressional defense committees by October 1, 2009. This report should include, at minimum, * an explanation of the cost and other risks associated with a cost- reimbursable contract as compared to a fixed-price contract for JSF's future low-rate production quantities, * the program's strategy for managing and mitigating risks associated with the use of cost contracts, and: * plans for transitioning to fixed-price contracts for production to include time frames and criteria. 2. Direct the JSF Program Office to ensure that the prime contractor performs periodic schedule risk analyses for the JSF program to provide better insight into management reserve, production efficiencies, and schedule completion to allow for corrections as early as possible. Agency Comments and Our Evaluation: DOD provided us with written comments on a draft of this report. The comments are reprinted in appendix III. DOD substantively agreed with our first recommendation regarding a report to the Congress on contracting strategy, but believed that the Under Secretary of Defense for Acquisition, Technology and Logistics should be responsible for the report, not the JSF Program Office as stated in our draft. As the milestone decision authority, the Under Secretary is responsible for approving the contracting strategy, contract awards, and the transition to full rate production. We agree with DOD and revised the recommendation accordingly. DOD also agreed with the second recommendation and will direct that the prime contractor perform periodic schedule risk analysis. In coordination with the JSF Program Office, the department intends to determine an optimum schedule for the contractor that will provide insight into JSF cost and schedule to influence key milestones and decision making. We are sending copies of this report to the Secretary of Defense; the Secretaries of the Air Force, Army, and Navy; and the Director of the Office of Management and Budget. The report also is available at no charge on the GAO Web site at [hyperlink, http://www.gao.gov]. If you or your staff have any questions concerning this report, please contact me at (202) 512-4841 or sullivanm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Staff members making key contributions to this report are listed in appendix IV. Signed by: Michael J. Sullivan: Director: Acquisition and Sourcing Management: List of Congressional Committees: The Honorable Carl Levin: Chairman: The Honorable John McCain: Ranking Member: Committee on Armed Services: United States Senate: The Honorable Daniel K. Inouye: Chairman: The Honorable Thad Cochran: Ranking Member: Subcommittee on Defense: Committee on Appropriations: United States Senate: The Honorable Ike Skelton: Chairman: The Honorable John M. McHugh: Ranking Member: Committee on Armed Services: House of Representatives: The Honorable John P. Murtha: Chairman: The Honorable C.W. Bill Young: Ranking Member: Subcommittee on Defense: Committee on Appropriations: House of Representatives: [End of section] Appendix I: Scope and Methodology: To determine the Joint Strike Fighter (JSF) program's progress in meeting cost, schedule, and performance goals, we received briefings by program and contractor officials and reviewed financial management reports, budget documents, annual selected acquisition reports, monthly status reports, performance indicators, and other data. We compared reported progress with prior years' data, identified changes in cost and schedule, and obtained officials' reasons for these changes. We interviewed Department of Defense (DOD), JSF Program Office, and contractor officials to obtain their views on progress, ongoing concerns and actions taken to address them, and future plans to complete JSF development and accelerate procurement. This review was the fifth and final effort under the mandate of the Ronald W. Reagan National Defense Authorization Act for Fiscal Year 2005. We were provided sufficient information to make the assessments contained in this report. In assessing program cost estimates, we compared the official program cost estimate in the December 31, 2007, selected acquisition report to estimates developed by the JSF program and an independent joint estimating team for fiscal years 2010 through 2015. Because the fiscal year 2010 budget had not been submitted to the Congress at the time of the draft report, some of the report's findings are based on preliminary cost projections that existed at the time of our review rather than the official program of record. As such, the cost projections in this report may be different than the final fiscal year 2010 program of record. We interviewed members of the joint estimating team to obtain a detailed understanding of the methodology, data, and approach used in developing their cost estimate and schedule risk analysis of the JSF program. We also interviewed JSF program officials to understand the program's cost estimate methodology, assumptions, and results and to obtain their response to the joint estimating team's analysis. Based on this analysis, we were able to identify significant differences in the cost estimating methodologies and assumptions used by the joint estimating team and the program office and to determine major risk drivers in the program. To assess the program's plans and risks in manufacturing and its capacity to accelerate production from fiscal years 2010 through 2015, we analyzed manufacturing cost and work performance data to assess progress against plans. We compared budgeted program labor hours to actual labor hours, identified growth trends, and noted differences between future labor requirements and current plans to release engineering staff. We reviewed data and briefings provided by the program and the Office of the Secretary of Defense (OSD) to assess supplier performance and ability to support accelerated production from fiscal years 2010 through 2015. We also determined reasons for manufacturing delays, discussed program and contractor plans to improve, and projected expected impacts on development and operational tests. We also reviewed the prime contractor's schedule estimates and compared them with relevant best practices[Footnote 14] to determine the extent to which they reflect key estimating practices that are fundamental to having a reliable schedule. In doing so, we interviewed cognizant program officials to discuss their use of best practices in creating the program's current schedule and interviewed officials from Naval Air Systems Command to understand their approach and to obtain results of their independent schedule risk analysis. To assess plans, progress, and risks in test activities, we examined program documents and interviewed DOD, program office, and contractor officials about current test plans and progress. To assess progress toward test plans, we compared the number of flight tests conducted as of October 2008 to those in the original test plan established in 2007. We also reviewed documents and interviewed prime contractor officials regarding flight testing, the integrated airborne test bed, and ground testing. To further assess the ground labs and test bed, we interviewed DOD and program officials and toured the testing labs and aircraft at the Lockheed Martin plant in Fort Worth, Texas. In performing our work, we obtained information and interviewed officials from the JSF Joint Program Office, Arlington, Virginia; Lockheed Martin Aeronautics, Fort Worth, Texas; Defense Contract Management Agency, Fort Worth, Texas; Defense Contract Management Agency, East Hartford, Connecticut; Naval Air Systems Command, Patuxent River, Maryland; Air Force Operational Test and Evaluation Center, Kirtland Air Force Base, New Mexico; Air Force Cost Analysis Agency, Arlington, Virginia; and OSD offices of the Under Secretary of Defense for Acquisition, Technology and Logistics, the Director of Program Analysis and Evaluation and its Cost Analysis Improvement Group, and the Director of Operational Test and Evaluation in Washington, D.C. We conducted this performance audit from June 2008 to March 2009 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. [End of section] Appendix II: GAO Assessment of Prime Contractor Schedule Management Processes: Prime Contractor's Scheduling Management Processes Meet Many Best Practices, but Program Risks Are Not Entirely Visible: The success of any program depends in part on having a reliable schedule of when the program's set of work activities will occur, how long they will take, and how they are related to one another. As such, the schedule not only provides a road map for systematic execution of a program, but also provides the means by which to gauge progress, identify and address potential problems, and promote accountability. In general, best practices and related federal guidance call for a program schedule to identify, sequence, integrate, and resource all key activities to be performed, and to understand and proactively address activities that pose critical risks. More specifically, our research has identified nine practices associated with effective schedule estimating.[Footnote 15] These practices are (1) capturing all activities, (2) sequencing all activities, (3) assigning resources to all activities, (4) establishing the duration of all activities, (5) integrating schedule activities horizontally and vertically, (6) establishing the critical path for all activities, (7) identifying float[Footnote 16] between activities, (8) conducting a schedule risk analysis, and (9) updating the schedule using logic and durations to determine dates. Of these nine practices, the JSF program either met or partially met eight with only one not being met. The area that did not meet the practices related to performing a schedule risk analysis. Specifically, the JSF program has not conducted its own schedule risk analysis that would determine the level of confidence it has in meeting completion dates. Further, an assessment is also critical to identifying and apportioning reserves for contingencies. Since the JSF program has not conducted its own schedule risk analysis, it has limited insight into areas of risk and uncertainty in the schedule. Naval Air Systems Command officials did accomplish an independent schedule risk analysis that indicated that the program could slip more than 2 years based on the productivity risks associated with software development and assembly of the various airframes as well as the time needed to complete all flight testing. In addition to a schedule risk analysis not being performed, we found several other schedule management concerns that further reduce the visibility of manufacturing risks. First, the use of best scheduling practices at the subcontractor level is still being developed, potentially affecting the integration of subcontractor schedules into the integrated master schedule. Integrating prime and subcontractor schedules is critical to meeting program schedules and cost expectations. The prime contractor is working with subcontractors to increase their level of schedule maturity. Another area of concern is that out-of-station work made it difficult to identify specific span times for individual manufacturing tasks.[Footnote 17] As a result, the detailed information related to the manufacturing work was not visible in the master schedule. Furthermore, because of the program's enormous size and complexity, the schedule has been difficult to maintain, requiring manual validation processes to ensure its integrity and validity. Ongoing JSF schedule validity will be an area that needs careful attention as it represents a potential weak point in the overall implementation of the integrated master schedule. Despite this shortcoming, it is also important to recognize the significant progress that the JSF program team has made in the area of schedule management. Since the previous Defense Contract Management Agency schedule review, both the schedule and the processes to manage it have greatly improved. For example, the schedule can track and verify activities associated with over 600 projects. It also successfully captures and sequences key activities and establishes the critical path between key activities, allowing the program to examine the impacts of schedule delays and determine schedule flexibility. The Sheer Size and Complexity of the JSF Schedule Have Created Major Challenges to Ensuring Schedule Integrity and Validity: The JSF schedule is maintained in Microsoft Project and consists of over 600 individual projects. Because the size and complexity of the schedule is so immense, it has been difficult to maintain. As such, a number of manual validation processes are required to ensure its integrity and validity. To its credit, the contractor has developed custom processes and tools to help manage the program schedule. However, because of its enormous size and complexity, the JSF's ongoing schedule validity will be an area that needs careful attention as it represents a potential weak point in the overall implementation of the integrated master schedule. Because the schedule was so large, we reviewed a subset of it, focusing on the delivery of one airframe for each variant of the F-35 being produced (i.e., BF4, AF1, and AF3). This subset schedule covered a time span from August 2006 through September 2014, and we analyzed it against our best practices for effective schedule estimating. See table 6 for the results of our analyses relative to each of the nine practices. Table 6: Prime Contractor Schedule Management System Compared to Best Practices: Criterion: Capturing all activities; Explanation: The schedule should reflect all activities (e.g., steps, events, outcomes, etc.) as defined in the program's work breakdown structure, to include activities to be performed by both the government and its contractors; Criterion met?[A]: Yes; GAO analysis: The program's schedule reflected both government and contractor activities, including activities related to designing, developing, testing, and manufacturing the three variants of the F-35. Criterion: Sequencing all activities; Explanation: The schedule should be planned so that it can meet program- critical dates. To meet this objective, key activities need to be logically sequenced in the order that they are to be carried out. In particular, activities that must finish prior to the start of other activities (i.e., predecessor activities) as well as activities that cannot begin until other activities are completed (i.e., successor activities) should be identified. By doing so, interdependencies among activities that collectively lead to the accomplishment of events or milestones can be established and used as a basis for guiding work and measuring progress; Criterion met?[A]: Yes; GAO analysis: The schedule included the sequencing of key activities, meaning that it included both the predecessor and successor activities and thus established interdependencies among the activities that form the basis for guiding work and measuring progress. The schedule logic was validated by numerous metrics that were tracked on a monthly basis. For example, the contractor collected metrics that verified that logic was not maintained at summary-level tasks, activities had both a predecessor and successor relationship, and external links were valid. Additionally, the metrics tracked the use of hard constraints so if any were found, there were procedures to remove any constraints that override schedule logic. These detailed metrics provided assurance in the validity of the schedule. They also enabled the program to have confidence that the sequencing of activities was not being overwritten by an overuse of constraints. Criterion: Assigning resources to all activities; Explanation: The schedule should reflect what resources (i.e., labor, material, and overhead) are needed to do the work, whether all required resources will be available when they are needed, and whether any funding or time constraints exist; Criterion met?[A]: Partially; GAO analysis: The resources, labor, materials, and overhead were not loaded directly into the Microsoft Project schedules. Instead, the budgets were maintained in another system, which was integrated with both the schedule and the Manufacturing Resource Planning (MRP) system. While not technically assigned to individual activities, the resources were integrated into the management of the system through the work packages identified in the earned value system. The manufacturing activity resources were maintained in the MRP system and the schedule reflected the level of work accomplished in various production stations based on the status of the individual work cards. Program officials provided a data trace between the schedule, MRP system, and earned value system to demonstrate how the resources were integrated with the schedule. Criterion: Establishing the duration of all activities; Explanation: The schedule should realistically reflect how long each activity will take to execute. In determining the duration of each activity, the same rationale, data, and assumptions used for cost estimating should be used for schedule estimating. Further, these durations should be as short as possible and they should have specific start and end dates. Excessively long periods needed to execute an activity should prompt further decomposition of the activity so that shorter execution durations will result; Criterion met?[A]: Partially; GAO analysis: The schedule established the durations of key activities based on historical data from the contractor's experience, projected savings to be gained from implementing robust processes to maintain the flow of work on the manufacturing floor, and expected savings from investing up front in modeling and simulation. The program tracked the actual start, progress, and actual finish of activities in the schedule, allowing for collection of trending information. Progress metrics were tracked by the contractor in the status metric process and reported to the program, enabling tracking against the schedules; It was difficult, however, to see the specific span times for individual manufacturing tasks because work that did not finish in its designated station was carried forward and completed at a different station down the production line (this work was called out-of-station). As a result, the detailed information related to the manufacturing work was maintained in the MRP system, but was not visible within the schedule itself. Nonetheless, the contractor demonstrated the relationship between the specific work cards in the MRP system and how those data translated to the schedule. Criterion: Integrating schedule activities horizontally and vertically; Explanation: The schedule should be horizontally integrated, meaning that it should link the products and outcomes associated with already sequenced activities. These links are commonly referred to as handoffs and serve to verify that activities are arranged in the right order to achieve aggregated products or outcomes. The schedule should also be vertically integrated, meaning that traceability exists among varying levels of activities and supporting tasks and subtasks. Such mapping or alignment among levels enables different groups to work to the same master schedule; Criterion met?[A]: Yes; GAO analysis: The schedule was both horizontally and vertically integrated and there was a robust process in place to verify the traceability at each progress cycle. The contractor's project status sheet identified where activities were missing relationships, enabling a critical path to be derived by the tool's critical path analysis. Relationships were maintained across the 600+ individual projects and verified through periodic schedule health checks. The contractor provided report outputs and demonstrated the process it used to identify and validate that the links across projects were intact. Similarly, for vertical integration the contractor demonstrated how status was rolled up from the individual project files to the proper higher-level summary activities so the overall schedule reflected the proper dates and status; An area of concern was with the integration of the subcontractors' schedules into the integrated master schedule. The issue was not as much the integration of the data, but the use of scheduling best practices at the subcontractor level. The contractor was working with its subcontractors to increase the level of scheduling maturity. Moreover, given the massive size of the integrated master schedule and number of files being utilized to integrate it, this was an area that needed continual monitoring to verify compliance. Criterion: Establishing the critical path for all activities; Explanation: Using scheduling software the critical path--the longest duration path through the sequenced list of key activities--should be identified. The establishment of a program's critical path is necessary for examining the effects of any activity slipping along this path. Potential problems that may occur on or near the critical path should also be identified and reflected in the scheduling of the time for high- risk activities; Criterion met?[A]: Yes; GAO analysis: The program's critical path was defined using scheduling software. Critical paths were defined by individual airframes and were also calculated at the total program level. The contractor provided output from the critical path analysis and demonstrated the process it used to produce the critical path. Given the large number of files and overall size of the integrated master schedule, it was not possible to validate the critical path through independent analysis; The contractor's metric process performed during each status cycle helped to maintain and validate the critical path produced by the tool. In the subset schedule we reviewed there were no hard constraints, that is, we found no "must start" or "must finish" on dates that could override the critical path analysis and cause invalid results. Criterion: Identifying float between activities; Explanation: The schedule should identify float--the time that a predecessor activity can slip before the delay affects successor activities--so that schedule flexibility can be determined. As a general rule, activities along the critical path typically have the least amount of float; Criterion met?[A]: Yes; GAO analysis: The contractor's overall schedule process enabled good visibility into the float between activities and demonstrated that float is actively managed. The metrics tracked by the contractor clearly identified where there were issues with the amount of float on an activity, including which tasks had negative float. The contractor had defined processes for tracing and understanding what was driving that float. In addition, the tracking and removal of hard constraints in the schedule enabled the tool to generate the float via the critical path analysis. Criterion: Conducting a schedule risk analysis; Explanation: A schedule risk analysis uses a good critical path method schedule and data about project schedule risks as well as Monte Carlo simulation techniques to predict the level of confidence in meeting a program's completion date, the amount of time contingency needed for a level of confidence, and the identification of high-priority risks. This analysis focuses not only on critical path activities but also on other schedule paths that may become critical. A schedule/cost risk assessment recognizes the interrelationship between schedule and cost and captures the risk that schedule durations and cost estimates may vary because of, among other things, limited data, optimistic estimating, technical challenges, and lack of qualified personnel. As a result, the baseline schedule should include a buffer or a reserve of extra time. Schedule reserve for contingencies should be calculated by performing a schedule risk analysis. As a general rule, the reserve should be held by the project manager and applied as needed to those activities that take longer than scheduled because of the identified risks. Reserves of time should not be apportioned in advance to any specific activity since the risks that will actually occur and the magnitude of their impact is not known in advance; Criterion met?[A]: No; GAO analysis: The contractor did not perform a schedule risk analysis that would have determined the level of confidence in meeting the program's completion date and identified reserves for contingencies because it did not have time to do this analysis. The contractor spent much of the past year rebaselining its schedule so it would be more realistic and meet best practices and said that it intended to do schedule risk analyses in the future; Naval Air Systems Command conducted an independent schedule risk analysis that indicated that the program could slip more than 2 years based on productivity risks associated with software development and assembly of the various airframes as well as the time needed to successfully complete all flight testing. Based on the command's results, the contractor does not have an adequate amount of schedule reserve to mitigate testing and production risks. This low reserve means that any delays in these areas would have an immediate impact to the overall project completion date. Moreover, there was concern from outside experts that savings gained from implementing robust manufacturing processes and relying more on modeling and simulation may not be achieved as they exceed anything done before. Both the contractor and the JSF Program Office disputed the findings of the command's schedule risk analysis primarily because the analysis was done without direct involvement of program officials. Criterion: Updating the schedule using logic and durations to determine the dates; Explanation: The schedule should use logic and durations in order to reflect realistic start and completion dates for program activities. The schedule should be continually monitored to determine when forecasted completion dates differ from the planned dates, which can be used to determine whether schedule variances will affect downstream work. Maintaining the integrity of the schedule logic is not only necessary to reflect true status, but is also required before conducting a schedule risk analysis. The schedule should avoid logic overrides and artificial constraint dates that are chosen to create a certain result on paper. To ensure that the schedule is properly updated, individuals trained in critical path method scheduling should be responsible for updating the schedule status; Criterion met?[A]: Yes; GAO analysis: The contractor demonstrated that it relies on logic and duration to determine the dates when it updates the status of its schedule. For example, the contractor showed us how it monitors the schedule on a weekly basis to ensure that start and completion dates are realistic and to identify where there are variances between planned and actual progress. The contractor also demonstrated how it verifies that all hard constraints have been removed so that the schedule depicts actual conditions. Moreover, the personnel responsible for maintaining the schedule demonstrated a strong understanding of critical path method scheduling, adding to our confidence that the schedule is being properly updated in accordance with best practices. Source: GAO analysis of DOD data. [A] "Yes" means that the program provided documentation demonstrating satisfaction of the criterion. "Partial" means that the program provided documentation demonstrating satisfaction of part of the criterion. "No" means that the program has yet to provide documentation demonstrating satisfaction of the criterion. [End of table] [End of section] Appendix III: Comments from the Department of Defense: Office Of The Under Secretary Of Defense: 3000 Defense Pentagon: Washington, DC 20301-3000: Acquisition Technology And Logistics: March 5, 2009: Mr. Michael Sullivan: Director, Acquisition and Sourcing Management: U.S. Government Accountability Office: 441 G Street, N.W.: Washington, DC 20548: Dear Mr. Sullivan: This is the Department of Defense (DoD) response to the GAO draft report 09-303, "Joint Strike Fighter: DOD'S Proposal to Accelerate Procurement before Completing Development Increases the Government's Financial Risk" dated February 2, 2009, (GAO Code 120742). Detailed comments on the report recommendations are enclosed. The DoD partially concurs with recommendation one and concurs with recommendation two. The rationale for our position is included in the enclosure. We appreciate the opportunity to comment on the draft report. My point of contact for this effort is CAPT Gregg Sears, 703-695-3015, gregg.sears@osd.mil. Sincerely, Signed by: David G. Ahern: Director: Enclosure: As stated: Portfolio Systems Acquisition GAO Draft Report Dated FEBRUARY 2, 2009 GAO-09-303 (GAO CODE 120742): "Joint Strike Fighter: Dod's Proposal To Accelerate Procurement Before Completing Development Increases The Government's Financial Risk" Department Of Defense: Comments To The GAO Recommendations: Recommendation 1: The GAO recommends that the Secretary of Defense direct that the Joint Strike Fighter program office report to the Congressional Defense Committees by October 1, 2009. DOD Response: Partially concur The Department agrees that a report to the Congressional Defense Committees explaining the Department's plan to transition from cost reimbursable contracts to fixed price contracts for Joint Strike Fighter (JSF) procurement would be beneficial to enhance Congressional oversight and confidence in the JSF program. However, the Department believes that the Under Secretary of Defense for Acquisition, Technology and Logistics (USD AT&L) should be responsible for the report, not the Joint Strike Fighter program. As the Milestone Decision Authority (MDA), USD (AT&L) is responsible for approval of the JSF Acquisition Strategy, including the contracting strategy, as well as approval of individual Low Rate Initial Production (LRIP) contract awards and the transition to Full Rate Production (FRP). The JSF Acquisition Strategy, utilizing cost reimbursable contracts that contain cost and performance incentives for LRIP, has been approved by the Department and involved a review of the associated risks, benefits, and options. The USD AT&L, on behalf of the Secretary will provide a report to the Congressional Defense Committees by October 1, 2009, that addresses the requirements in the recommendation. The Department would also like to state that since the FY 2010 President's budget had not yet been submitted to Congress at the time of the draft report, the fact is that much of the data that the GAO used as the basis of their report and to support its recommendation was pre- decisional. The JSF program of record remained as submitted in the FY 2009 President's Budget, adjusted by the FY 2009 Defense Appropriations Bill. The Department believes that the GAO's use of pre- decisional budget information to support its findings is largely conjectural, and likely to create confusion if the final FY 2010 program of record is different than what the GAO reported. The JSF program does have substantial concurrency of development, test and production built into the JSF schedule, a fact that the Department acknowledges and has approved. That concurrency is designed to provide the warfighters with a 5th generation strike fighter to replace aging legacy aircraft as quickly as possible, as efficiently as possible, and as affordably as possible. The Department acknowledges the risks, and benefits, of the concurrency required to acquire and deliver this critical capability. The Department also believes that the program is well managed, with the proper amount of oversight, and well positioned to succeed in accomplishing this mission. Recommendation 2: The GAO recommends that the Secretary of Defense direct that the Joint Program Office ensure the prime contractor performs periodic schedule risk analyses for the Joint Strike Fighter (JSF) program to provide better insight into Management Reserve, production efficiencies, and schedule completion to allow for corrections as early as possible. DOD Response: Concur The Department will direct that the contractor perform periodic schedule risk analysis. The Department will coordinate with the JSF Program Office and determine an optimum schedule for the contractor to meet that will provide appropriate insight into JSF cost and schedule to influence key program milestones and decision making. [End of section] Appendix IV: GAO Contact and Staff Acknowledgments: GAO Contact: Michael Sullivan (202) 512-4841 or sullivanm@gao.gov: Acknowledgments: In addition to the contact named above, the following staff members made key contributions to this report: Bruce Fairbairn, Assistant Director; Ridge Bowman; Charlie Shivers; Georgeann Higgins; Matt Lea; Karen Richey; Tim Boatwright; and Greg Campbell. [End of section] Related GAO Products: Defense Acquisitions: Better Weapon Program Outcomes Require Discipline, Accountability, and Fundamental Changes in the Acquisition Environment. [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-08- 782T]. Washington, D.C.: June 3, 2008. Defense Acquisitions: Assessments of Selected Weapon Programs. [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-08-467SP]. Washington, D.C.: March 31, 2008. Joint Strike Fighter: Impact of Recent Decisions on Program Risks. [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-08-569T]. Washington, D.C.: March 11, 2008. Joint Strike Fighter: Recent Decisions by DOD Add to Program Risks. [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-08-388]. Washington, D.C.: March 11, 2008. Tactical Aircraft: DOD Needs a Joint and Integrated Investment Strategy. [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-415]. Washington, D.C.: April 2, 2007. Defense Acquisitions: Assessments of Selected Weapon Programs. [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-406SP]. Washington, D.C.: March 30, 2007. Defense Acquisitions: Analysis of Costs for the Joint Strike Fighter Engine Program. [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07- 656T]. Washington, D.C.: March 22, 2007. Joint Strike Fighter: Progress Made and Challenges Remain. [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-360]. Washington, D.C.: March 15, 2007. Tactical Aircraft: DOD's Cancellation of the Joint Strike Fighter Alternate Engine Program Was Not Based on a Comprehensive Analysis. [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-06-717R]. Washington, D.C.: May 22, 2006. Defense Acquisitions: Major Weapon Systems Continue to Experience Cost and Schedule Problems under DOD's Revised Policy. [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-06-368]. Washington, D.C.: April 13, 2006. Defense Acquisitions: Actions Needed to Get Better Results on Weapons Systems Investments. [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO- 06-585T]. Washington, D.C.: April 5, 2006. Tactical Aircraft: Recapitalization Goals Are Not Supported by Knowledge-Based F-22A and JSF Business Cases. [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-06-487T]. Washington, D.C.: March 16, 2006. Joint Strike Fighter: DOD Plans to Enter Production before Testing Demonstrates Acceptable Performance. [hyperlink, http://www.gao.gov/cgi- bin/getrpt?GAO-06-356]. Washington, D.C.: March 15, 2006. Joint Strike Fighter: Management of the Technology Transfer Process. [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-06-364]. Washington, D.C.: March 14, 2006. Tactical Aircraft: F/A-22 and JSF Acquisition Plans and Implications for Tactical Aircraft Modernization. [hyperlink, http://www.gao.gov/cgi- bin/getrpt?GAO-05-519T]. Washington, D.C: April 6, 2005. Tactical Aircraft: Opportunity to Reduce Risks in the Joint Strike Fighter Program with Different Acquisition Strategy. [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-05-271]. Washington, D.C.: March 15, 2005. [End of section] Footnotes: [1] Pub. L. No. 108-375, § 213 (2004). [2] GAO, Joint Strike Fighter: Recent Decisions by DOD Add to Program Risks, [hyperlink, http://www.gao.gov/products/GAO-08-388] (Washington, D.C.: Mar. 11, 2008); Joint Strike Fighter: Progress Made and Challenges Remain, [hyperlink, http://www.gao.gov/products/GAO-07-360] (Washington, D.C.: Mar. 15, 2007); Joint Strike Fighter: DOD Plans to Enter Production before Testing Demonstrates Acceptable Performance, [hyperlink, http://www.gao.gov/products/GAO-06-356] (Washington, D.C.: Mar. 15, 2006); and Tactical Aircraft: Opportunity to Reduce Risks in the Joint Strike Fighter Program with Different Acquisition Strategy, [hyperlink, http://www.gao.gov/products/GAO-05-271] (Washington, D.C.: Mar. 15, 2005). [3] The international partners are the United Kingdom, Italy, the Netherlands, Turkey, Canada, Australia, Denmark, and Norway. These nations are contributing funds for system development and have signed agreements to procure a minimum of 730 aircraft. Israel and Singapore are security cooperation participants, and several other nations have reportedly expressed interest in acquiring aircraft. [4] [hyperlink, http://www.gao.gov/products/GAO-08-388]. [5] In April 2008, OSD asked the Cost Analysis Improvement Group (CAIG) to lead a joint estimating team in assessing the overall executability of the JSF development program and resource requirements for fiscal years 2010 through 2015. The joint estimating team was composed of CAIG, Air Force, and Navy representatives as well as subject matter experts. [6] Management reserves are budgeted funds set aside for unanticipated development challenges and increase a program's capacity to deal with unknowns. [7] [hyperlink, http://www.gao.gov/products/GAO-08-388]. [8] The difference in the two estimates is actually more than $5 billion. The joint estimating team's numbers only go through fiscal year 2015, but they are expecting development to take at least 1 more year beyond then. Program data presented to the Joint Chiefs suggest that an additional year for development flight testing and other government and contractor expenses could cost about $700 million. [9] Federal Acquisition Regulation § 16.301-2. [10] An efficient production line establishes an orderly flow of work as a product moves from workstation to workstation and on to final assembly. Out-of-station work, sometimes referred to as traveled work, refers to completing unfinished work on major components, for example, the wings, after they have left the wing workstation and moved down the production line to another station, such as mate and final assembly. [11] GAO, GAO Cost Estimating and Assessment Guide: Best Practices for Developing and Managing Capital Program Costs, [hyperlink, http://www.gao.gov/products/GAO-09-3SP] (Washington, D.C.: March 2009). Also, see app. II for our assessment of the prime contractor's schedule management processes. [12] [hyperlink, http://www.gao.gov/products/GAO-08-388]. [13] GAO's past work has shown that manufacturing maturity is achieved when a company can produce a product within cost, schedule, and quality targets. A best practice is to ensure that all key manufacturing processes are repeatable, sustainable, and capable of consistently producing parts within the product's quality tolerances and standards. [14] GAO, Cost Assessment Guide: Best Practices for Estimating and Managing Program Costs, Exposure Draft, [hyperlink, http://www.gao.gov/products/GAO-07-1134SP] (Washington, D.C.: July 2007). [15] [hyperlink, http://www.gao.gov/products/GAO-09-3SP]. [16] Float is the amount of time an activity can slip before affecting the critical path. [17] It was difficult to see specific span times for individual manufacturing tasks because work that did not finish in its designated station was carried forward and completed at a different station down the production line. This work was called out-of-station.