This is the accessible text file for GAO report number GAO-14-322 entitled 'F-35 Joint Strike Fighter: Problems Completing Software Testing May Hinder Delivery of Expected Warfighting Capabilities' which was released on March 24, 2014. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. United States Government Accountability Office: GAO: Report to Congressional Committees: March 2014: F-35 Joint Strike Fighter: Problems Completing Software Testing May Hinder Delivery of Expected Warfighting Capabilities: GAO-14-322: GAO Highlights: Highlights of GAO-14-322, a report to congressional committees. Why GAO Did This Study: The F-35 Lightning II, also known as the Joint Strike Fighter, is DOD's most costly and ambitious acquisition program. The program seeks to develop and field three aircraft variants for the Air Force, Navy, and Marine Corps and eight international partners. The F-35 is integral to U.S. and international plans to replace existing fighter aircraft and support future combat operations. Total U.S. planned investment in the F-35 program is approaching $400 billion to develop and acquire 2,457 aircraft through 2037, plus hundreds of billions of dollars in long-term spending to operate and maintain the aircraft. The National Defense Authorization Act for Fiscal Year 2010 mandated that GAO review the F-35 acquisition program annually for 6 years. In this, GAO's fifth annual report on the F-35, GAO assesses the program's (1) ongoing development and testing, (2) long-term affordability, and (3) manufacturing progress. GAO reviewed and analyzed manufacturing data through December 2013, program test plans, and internal DOD analyses, and spoke with DOD, program, and contractor officials. What GAO Found: Delays in developmental flight testing of the F-35's critical software may hinder delivery of the warfighting capabilities the military services expect. F-35 developmental flight testing comprises two key areas: mission systems and flight sciences. Mission systems testing verifies that the software-intensive systems that provide critical warfighting capabilities function properly and meet requirements, while flight sciences testing verifies the aircraft's basic flying capabilities. Challenges in development and testing of mission systems software continued through 2013, due largely to delays in software delivery, limited capability in the software when delivered, and the need to fix problems and retest multiple software versions. The Director of Operational Test and Evaluation (DOT&E) predicts delivery of warfighting capabilities could be delayed by as much as 13 months. Delays of this magnitude will likely limit the warfighting capabilities that are delivered to support the military services' initial operational capabilities—the first of which is scheduled for July 2015—-and at this time it is not clear what those specific capabilities will be because testing is still ongoing. In addition, delays could increase the already significant concurrency between testing and aircraft procurement and result in additional cost growth. Without a clear understanding of the specific capabilities that will initially be delivered, Congress and the military services may not be able to make fully informed resource allocation decisions. Flight sciences testing has seen better progress, as the F-35 program has been able to accomplish nearly all of its planned test flights and test points. Testing of the aircraft's operational capabilities in a realistic threat environment is scheduled to begin in 2015. The program has continued to make progress in addressing some key technical risks. To execute the program as planned, the Department of Defense (DOD) will have to increase funds steeply over the next 5 years and sustain an average of $12.6 billion per year through 2037; for several years, funding requirements will peak at around $15 billion. Annual funding of this magnitude clearly poses long-term affordability risks given the current fiscal environment. The program has been directed to reduce unit costs to meet established affordability targets before full-rate production begins in 2019, but meeting those targets will be challenging as significant cost reductions are needed. Additionally, the most recent cost estimate for operating and supporting the F-35 fleet is more than $1 trillion, which DOD officials have deemed unaffordable. This estimate reflects assumptions about key cost drivers the program can control, like aircraft reliability, and those it cannot control, including fuel costs, labor costs, and inflation rates. Reliability is lower than expected for two variants, and DOT&E reports that the F-35 program has limited additional opportunities to improve reliability. Aircraft manufacturing continued to improve in 2013, and management of the supply chain is evolving. As the number of aircraft in production has increased, critical learning has taken place and manufacturing efficiency has improved. For example, the prime contractor has seen reductions in overall labor hours needed to manufacture the aircraft, as expected. In 2013, the contractor delivered 35 aircraft to the government, 5 more than it delivered in 2012 and 26 more than it delivered in 2011. The prime contractor has put in place a supplier management system to oversee key supplier performance. What GAO Recommends: GAO recommends that DOD assess and identify the specific capabilities that realistically can be delivered to the military services to support their respective initial operational capabilities, and share its findings with the Congress and military services prior to July 2015. DOD concurred with this recommendation. View [hyperlink, http://www.gao.gov/products/GAO-14-322]. For more information, contact Mike Sullivan at (202) 512-4841 or sullivanm@gao.gov. [End of section] Contents: Letter: Background: Testing of Critical Software Poses a Significant Challenge to the Delivery of F-35 Warfighting Capabilities: Program Funding Projections and Unit Cost Targets May Not Be Achievable: F-35 Aircraft Manufacturing Efficiency Continues to Improve: Conclusions: Recommendation for Executive Action: Agency Comments: Appendix I: Prior GAO Reports and DOD Responses: Appendix II: Scope and Methodology: Appendix III: Comments from the Department of Defense: Appendix IV: GAO Contact and Staff Acknowledgments: Related GAO Products: Tables: Table 1: Changes in Reported F-35 Program Cost and Quantity and Deliveries 2001-2013: Table 2: Comparison of 2013 Unit Costs and 2019 Unit Cost Goals (dollars in then year millions): Table 3: Cost Drivers That Affect F-35 Operation and Sustainment Cost Estimates: Figures: Figure 1: F-35 Test Points Progress in 2013: Figure 2: Budgeted Development and Procurement Costs by Service, 2014- 2037 (dollars in then year billions): Figure 3: Relationship among Readiness, Reliability, and Sustainment Costs: Figure 4: F-35 Average Flying Hours between Failure Rates by Variant: Figure 5: Average Labor Hours Per Aircraft Since Beginning Low-Rate Production and 2014 Goal: Figure 6: Trend of F-35 Actual Delivery Dates Compared to Contractual Delivery Dates: Figure 7: Countries Currently Participating in F-35 Production: Abbreviations: ALIS: Autonomic Logistics Information System: CAPE: Cost Assessment and Program Evaluation: CTOL: conventional takeoff and landing: CV: carrier-suitable variant: DCMA: Defense Contract Management Agency: DOD: Department of Defense: DOT&E: Director of Operational Test and Evaluation: IOC: initial operational capability: STOVL: short takeoff and vertical landing: [End of section] United States Government Accountability Office: GAO: 441 G St. N.W. Washington, DC 20548: March 24, 2014: Congressional Committees: With estimated acquisition costs approaching $400 billion, the F-35 Lightning II--also known as the Joint Strike Fighter--is the Department of Defense's (DOD) most costly and ambitious acquisition program. The program is developing and fielding a family of next generation fighter aircraft, incorporating low observable (stealth) technologies as well as advanced sensors and computer networking capabilities for the United States Air Force, Navy, and Marine Corps as well as eight international partners.[Footnote 1] The F-35 family is comprised of three aircraft variants: (1) a conventional takeoff and landing (CTOL) variant, (2) a short takeoff and vertical landing (STOVL) variant, and (3) a carrier-suitable variant (CV). The F-35 is integral to U.S. and partner plans to replace existing fighter aircraft and support future combat operations. According to current plans, the U.S. portion of the program will require annual acquisition funding of more than $12 billion on average through 2037 to complete development and procure a total of 2,457 aircraft. In addition, the F- 35 fleet is estimated to cost around $1 trillion to operate and support over its lifetime. In a time of austere federal budgets, cost projections of this magnitude pose significant fiscal challenges. We have reported on F-35 issues for many years.[Footnote 2] Over time we have reported significant cost, schedule, and performance problems and have found that those problems, in large part, can be traced to (1) decisions made at key junctures without adequate product knowledge; and (2) a highly concurrent acquisition strategy with significant overlap among development, testing, and manufacturing activities. We have made numerous recommendations aimed at addressing these issues, and DOD has taken action to address them to varying degrees.[Footnote 3] In March 2012, DOD completed an extensive restructuring of the F-35 program and we reported in June 2012 and again in March 2013 that the restructuring actions should lead to more achievable and predictable outcomes, albeit at higher cost and longer timeframes for testing and delivering capabilities to the warfighter than originally planned.[Footnote 4] The National Defense Authorization Act for Fiscal Year 2010 mandated GAO to review the F-35 acquisition program annually for 6 years. [Footnote 5] This is our fifth report under that mandate, and in it, we assess the program's (1) ongoing development and testing, (2) funding and long-term affordability, and (3) manufacturing progress. To conduct our work, we reviewed and analyzed program briefings, management reports, program test results, and internal DOD program analyses. We discussed key aspects of F-35 performance with both military and private contractor test pilots. We interviewed F-35 program and aircraft prime contractor officials to discuss developmental testing. We also collected developmental test plans, and data on test achievements to assess program progress through December 2013. We obtained current program acquisition and life-cycle sustainment cost estimates, reviewed the supporting documentation and discussed the development of those estimates with DOD and prime contractor officials instrumental in producing them. We toured F-35 manufacturing and test facilities and obtained and analyzed production and supply chain data as of December 2013. We assessed the reliability of DOD and contractor data by reviewing existing information about the data, and interviewing agency officials knowledgeable about the data. We determined that the data were sufficiently reliable for the purposes of this report. We also discussed ongoing manufacturing process improvements with prime contractor and Defense Contract Management Agency (DCMA) officials. Appendix II contains a more detailed description of our scope and methodology. We conducted this performance audit from June 2013 to March 2014 in accordance with generally accepted government auditing standards. Those standards required that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Background: As we have reported in the past, DOD began the F-35 acquisition program--also known as the Joint Strike Fighter--in October 2001, without adequate knowledge about the aircraft's critical technologies or its design.[Footnote 6] In addition, the program's acquisition strategy called for high levels of concurrency between development, testing, and production. As a result, the program encountered significant cost and schedule growth as well as performance shortfalls and was restructured three times: first in December 2003, then again in March 2007, and most recently in March 2012. The most recent restructuring was initiated in early 2010, when the program's unit cost estimates exceeded critical thresholds established by statute--a condition known as a Nunn-McCurdy breach. DOD subsequently certified to the Congress in June 2010 that the program was essential to national security and needed to continue.[Footnote 7] DOD then began efforts to significantly restructure the program and establish a new acquisition program baseline. These restructuring efforts continued through 2011 and into 2012, during which the department increased the program's cost estimates, extended its testing and delivery schedules, and reduced near-term aircraft procurement quantities by deferring the procurement of 410 aircraft into the future. The new F-35 acquisition program baseline was finalized in March 2012, and since that time costs have remained relatively stable. Table 1 notes the significant cost, quantity, and schedule changes from the original program baseline and the relative stability since the new baseline was established. Table 1: Changes in Reported F-35 Program Cost and Quantity and Deliveries 2001-2013: Expected quantities (number of aircraft): Developmental quantities; October 2001 (initial baseline): 14; March 2012 (restructured baseline): 14; March 2013 (current estimates): 14; Change 2001-2012: 0%; Change 2012-2013: 0%. Procurement quantities (U.S. only); October 2001 (initial baseline): 2,852; March 2012 (restructured baseline): 2,443; March 2013 (current estimates): 2,443; Change 2001-2012: -14%; Change 2012-2013: 0%. Total quantities; October 2001 (initial baseline): 2,866; March 2012 (restructured baseline): 2,457; March 2013 (current estimates): 2,457; Change 2001-2012: -14%; Change 2012-2013: 0%. Cost estimates (then-year dollars in billions): Development; October 2001 (initial baseline): $34.4; March 2012 (restructured baseline): $55.2; March 2013 (current estimates): $55.2; Change 2001-2012: 60%; Change 2012-2013: 0%. Procurement; October 2001 (initial baseline): $196.6; March 2012 (restructured baseline): $335.7; March 2013 (current estimates): $330.6; Change 2001-2012: 71%; Change 2012-2013: -2%. Military construction; October 2001 (initial baseline): $2.0; March 2012 (restructured baseline): $4.8; March 2013 (current estimates): $4.6; Change 2001-2012: 140%; Change 2012-2013: -4%. Total program acquisition; October 2001 (initial baseline): $233.0; March 2012 (restructured baseline): $395.7; March 2013 (current estimates): $390.4; Change 2001-2012: 70%; Change 2012-2013: -1%. Unit cost estimates (then-year dollars in millions): Program acquisition; October 2001 (initial baseline): $81; March 2012 (restructured baseline): $161; March 2013 (current estimates): $159; Change 2001-2012: 99%; Change 2012-2013: -1%. Average procurement; October 2001 (initial baseline): $69; March 2012 (restructured baseline): $137; March 2013 (current estimates): $135; Change 2001-2012: 99%; Change 2012-2013: -1%. Estimated delivery and production dates: Initial operational capability; October 2001 (initial baseline): 2010-2012; March 2012 (restructured baseline): TBD; March 2013 (current estimates): 2015-2018; Change 2001-2012: NA; Change 2012-2013: NA. Full-rate production; October 2001 (initial baseline): 2012; March 2012 (restructured baseline): 2019; March 2013 (current estimates): 2019; Change 2001-2012: 7 years; Change 2012-2013: 0 years. Source: GAO analysis of DOD data. [End of table] At the time the new F-35 acquisition program baseline was finalized, it did not identify new initial operational capability (IOC) dates for the three military services.[Footnote 8] The following year DOD issued a memorandum noting that Marine Corps and Air Force were planning to field initial operational capabilities in 2015 and 2016, respectively, and that the Navy planned to field its initial capability in 2018. The memorandum emphasized that the Marine Corps and Air Force initial operational capabilities would be achieved with aircraft that possess initial combat capabilities, and noted that those aircraft would need additional lethality and survivability enhancements to meet the full spectrum of warfighter requirements in the future. These new parameters represented a delay of 5 to 6 years from the program's initial 2001 baseline and a reduction in the capabilities expected at IOC. In March 2005 we recommended that DOD implement an evolutionary, incremental approach to developing and fielding the F-35--then known as the Joint Strike Fighter--to ensure that the warfighters would receive an initial combat capability that, at a minimum, would meet their most immediate needs.[Footnote 9] Again in March 2010, we recommended that DOD identify the absolute minimum combat capabilities that would be acceptable by each of the military services to field their initial operational capabilities and establish reasonable, realistic timeframes for achieving those requirements.[Footnote 10] In both instances, we noted that the military services should consider trading off desired capabilities in order to more rapidly field aircraft with an initial set of usable capabilities and that any capabilities not needed to meet immediate warfighting needs should be deferred to a future development increment. In both instances, DOD agreed with the intent of our recommendation, but believed that its existing program management practices were sufficient. Testing of Critical Software Poses a Significant Challenge to the Delivery of F-35 Warfighting Capabilities: Delays in the testing of critical mission systems software have put the delivery of expected warfighting capabilities to the Marine Corps at risk, and could affect the delivery of capabilities to the Air Force and Navy as well. F-35 developmental flight testing is separated into two key areas: mission systems and flight sciences. Mission systems testing is done to verify that the software and systems that provide critical warfighting capabilities function properly and meet requirements, while flight science testing is done to verify the aircraft's basic flying capabilities. In a March 2013 report we found that development and testing of mission systems software was behind schedule, due largely to delayed software deliveries, limited capability in the software when delivered, and the need to fix problems and retest multiple software versions. These same challenges continued thorough 2013, and as a result progress in mission systems testing has been limited. The Director of Operational Test and Evaluation (DOT&E) predicts that the delivery of expected warfighting capabilities to the Marine Corps could be delayed by as much as 13 months. Delays of this magnitude could also increase the already significant concurrency between testing and aircraft procurement and result in additional cost growth. Although mission systems testing is behind, the F-35 program has been able to accomplish nearly all of its planned flight science testing. The program also continued to make progress in addressing key technical risks, although some of that progress has been limited. Limited Progress in Mission Systems Software Testing May Cause Delays and Add Cost to F-35 Development: While the F-35 program was able to accomplish all of the mission system test flights it had planned in 2013, it did not accomplish all of the planned test points,[Footnote 11] falling short by 11 percent. The F-35 program planned to fly 329 mission systems test flights and accomplish 2,817 test points in 2013. The program actually flew 352 test flights, exceeding the goal, but only accomplished 2,518 test points. According to program and contractor officials, slow progress in developing, delivering, and testing mission systems software continues to be the program's most significant risk area. The F-35 program is developing and fielding mission systems software capabilities in blocks: (1) Block 1, (2) Block 2A, (3) Block 2B, (4) Block 3i, and (5) Block 3F. Each subsequent block builds on the capabilities provided in the preceding blocks. Blocks 1 and 2A provide training capabilities and are essentially complete, with some final development and testing still underway. Blocks 2B and 3i provide initial warfighting capabilities and are needed by the Marine Corps and Air Force, respectively, to achieve initial operational capability. Block 3F is expected to provide the full suite of warfighting capabilities, and is the block the Navy expects to have to achieve its initial operational capability. Developmental testing of Block 2B software is behind schedule and will likely delay the delivery of expected warfighting capabilities. The delivery of this software capability is of high near-term importance because it provides initial warfighting capability for the overall F- 35 program, and is needed by the Marine Corps to field its initial operational capability in July 2015. As of January 2014, the program planned to have verified the functionality of 27 percent of the software's capability on-board the aircraft, but had only been able to verify 13 percent. This leaves a significant amount of work to be done before October 2014, which is when the program expects to complete developmental flight testing of this software block. According to DOT&E, Block 2B developmental testing will not be completed as scheduled and could be delayed by as much as 13 months, as the program has had to devote time and resources to address problems and completing development of prior software blocks. Delays of this magnitude would mean that the Marine Corps will not likely have all of the capabilities it expects in July 2015, its planned initial operational capability date. At this time it is not clear exactly which of the expected capabilities will be available as testing is still ongoing. The effects of these delays compound as they also put the timely delivery of Air Force and Navy initial operational capabilities at risk. The Air Force expects to field its initial operational capability in August 2016 with Block 3i aircraft that possess the same warfighting capabilities as the Marine Corps aircraft, but have upgraded computer hardware. The Navy plans to achieve its initial operational capability in August 2018 with Block 3F aircraft that possess the full suite of F-35 warfighting capabilities. Program and contractor officials have stated that while they recognize that the program faces software risks, they still expect to deliver all of the planned F-35 software capabilities to the military services as currently scheduled. However, given the uncertainty in mission systems software testing and the significance of the F-35 to future force structure plans, it is important that the military services have a clear understanding of the specific capabilities that they can realistically expect to receive, and those capabilities that are not likely to be delivered by their initial operational capability dates-- the first of which is scheduled for July 2015. In addition, because the F-35 is DOD's most costly and ambitious acquisition program, it is important that DOD keep Congress informed about the status of the program. Without a clear understanding of the specific capabilities that will initially be delivered, Congress and the military services may not be able to make fully informed resource allocation decisions. Delays in mission systems software testing could also increase costs. As currently planned, DOD expects to complete developmental flight testing in 2017. If the flight test schedule is extended, the program may have to retain testing and engineering personnel longer than currently expected, which would increase development cost. DOD currently expects to have invested $70.2 billion to procure 359 aircraft by 2017 when developmental flight testing is scheduled to end. Our past reports have concluded that purchasing aircraft while concurrently conducting developmental flight testing increases the risk that problems will be discovered late in testing and additional funding will be needed to rework aircraft that have already been purchased.[Footnote 12] If F-35 procurement plans remain unchanged and developmental testing continues into 2018, the cost risks associated with concurrency will likely increase as DOD expects to have invested $83.4 billion in 459 aircraft by that point in time. The F-35 contractor recognizes that additional testing efficiencies are important in order to deliver capabilities on schedule and cost. One way it plans to gain efficiency is to use test results from one F- 35 variant to close out test points for the other two variants in instances in which the variants have common functions. According to test officials, most mission systems testing can be accomplished on any variant and only a limited amount of variant-specific testing is required. Contractor officials pointed out that this type of efficiency would help mitigate some testing risk, but they also recognized that it will still be difficult to make up the lost time in the test program. They noted that delays in specific test events generally impact the entire test schedule because the ability to conduct future testing is often dependent on the completion of the earlier events. Program Accomplished Most Flight Science Test Goals in 2013: The program accomplished nearly all of the flight sciences testing, including weapons testing, it had planned for 2013. As of December 2013, the program had achieved half of the total number of test points required to complete all flight science testing for the program. Figure 1 below shows the number of flight science test points planned and accomplished for each F-35 variant as of December 2013, and also identifies the total number of test points required for each variant to complete flight science testing. Figure 1: F-35 Test Points Progress in 2013: [Refer to PDF for image: horizontal bar graph] Carrier variant flight sciences: Test points cumulative planned 2013: 5,477; Test points cumulative performed 2013: 6,241; Total test points currently expected: 14,646. Short take-off vertical landing flight sciences: Test points cumulative planned 2013: 9,500; Test points cumulative performed 2013: 9,886; Total test points currently expected: 19,988. Conventional take-off and landing flight sciences: Test points cumulative planned 2013: 8,878; Test points cumulative performed 2013: 8,212, Total test points currently expected: 13,985. Source: GAO analysis of DOD data. [End of figure] The program made progress despite the fact that flight testing was halted twice at the beginning of the year to investigate and fix cracks in an engine fan blade and leaky fuel hoses. In addition, program and contractor officials emphasized that employee furloughs that occurred in 2013, due to mandatory sequestration, limited the amount of flight testing that could be done during that time as well. [Footnote 13] Some of the key flight science and weapons testing accomplishments included: * Conventional takeoff and landing variant - The program successfully demonstrated the variant's ability to launch AIM-120 missiles from its internal weapons bay and to refuel while in flight. The program also continued testing the aircraft's ability to function at high vertical flight angles, although program officials noted that the testing took longer than expected. As of December 2013, the program had accomplished 59 percent of its total expected flight science test points for this variant. * Short takeoff and vertical landing variant - The program successfully demonstrated the STOVL's ability to takeoff vertically, launch weapons from its internal weapons bay, and dump fuel when needed. In addition, the program conducted some testing of the variant at sea on an amphibious assault ship--specifically the USS WASP. As of December 2013, the program had accomplished 49 percent of its total expected flight science test points for this variant. * Carrier-suitable variant - The program began testing the capability of the aircraft to function at high vertical flight angles. In addition, the program successfully demonstrated the aircraft's ability to dump fuel when needed. Program and contractor officials noted that the program also made progress to begin testing to verify that the aircraft's new arresting hook system could successfully catch a cable on a set of carrier arresting gear installed onshore at the Lakehurst facility. As of December 2013, the program had accomplished 43 percent of its total expected flight science test points for this variant. Developmental testing is not the only testing that the program still has to complete. The F-35 program is also scheduled to begin operational testing in June 2015, to determine that the aircraft variants can effectively perform their intended missions in a realistic threat environment.[Footnote 14] Program Progress in Addressing Key Technical Risks Has Been Mixed: While the F-35 program made progress addressing some key technical risks in 2013, it continued to encounter slower than expected progress in developing the Autonomic Logistics Information System (ALIS). Over time, we have reported on 4 areas of technical and structural risk that the program identified during flight, ground, and lab testing that if not addressed, could result in substantially degraded capabilities and mission effectiveness.[Footnote 15] In 2013, we found that the program made the following progress in each of those areas: * Helmet mounted display - provides flight data, targeting, and other sensor data to the pilot, and is integral to reducing pilot workload and achieving the F-35's concept of operations. The original helmet mounted display encountered significant technical deficiencies, including display jitter, the undesired shaking of the visor display, and latency, the perceivable lag that occurs in transmitting sensor data, and did not meet warfighter requirements. The program made adjustments to the helmet design, including adding sensors to lessen the display jitter, and redesigning elements to minimize latency. The program tested these design changes in 2013 and found that most of the technical deficiencies had been adequately addressed, and that the helmet's performance was sufficiently suitable to support Marine Corps initial operational capability in 2015. DOT&E and program test pilots noted that the current night vision camera continues to have problems. The program has identified a new camera that it believes will address those problems, but that camera has not been fully tested to verify its capabilities. * Arresting Hook System - allows the F-35 carrier-suitable variant to engage landing wires on aircraft carriers, was redesigned after the original hook system was found to be deficient. The program determined that the original hook assembly was not strong enough to reliably catch the wire and stop the airplane. As a result, the program modified the hook system's hydraulic components, and made structural modifications to the plane. In March 2013, the program completed a critical design review of the hook system to verify that the new design is sound. Land testing of the redesigned system has been successful, and the program anticipates that it will be ready for carrier testing in October 2014. * Durability - structural and durability testing of the aircraft continued in 2013, and the program completed the first round of this testing on all three variants. The conventional takeoff and landing variant and the short takeoff and vertical landing variants have also started their second round of testing. During this second round of testing, the short takeoff and vertical landing test aircraft developed bulkhead cracks at the equivalent of 17 years of service life. Contractor officials noted that they were working to develop a solution to those cracks, but the total cost and schedule impacts of these bulkhead cracks are unknown at this time. * Autonomic Logistics Information System - an important tool to predict and diagnose maintenance and supply issues, automate logistics support processes and provide decision aids aimed at reducing life- cycle sustainment costs and improving force readiness. ALIS is being developed and fielded in increments. In 2013, the program had to release an update to the first increment because problems were discovered after the increment was released to the testing locations. The additional time to develop and field this update will likely delay the delivery of future increments. The program completed site activation of ALIS systems at some training and testing locations, and is in the process of adding capabilities and maturing ALIS in a second increment to support the Marine Corps' initial operational capability. DOT&E notes that, although the second increment is scheduled to be delivered in time to support the Marine Corps' initial operational capability, there is no margin for error in the development schedule. Testing of this ALIS increment is about two months behind largely due to a lack of test facilities. Program officials note that they are in the process of adding facilities. The third, and final, increment of ALIS that provides full capability is not expected to be released until 2016. Program Funding Projections and Unit Cost Targets May Not Be Achievable: The F-35 program's high projected annual acquisition funding levels continue to put the program's long-term affordability at risk. Currently the acquisition program requires $12.6 billion per year through 2037, which does not appear to be achievable given the current fiscal environment. The program is reducing unit costs to meet targets, but a significant amount of additional cost reduction is needed if it expects to meet those targets before the beginning of full rate production--currently scheduled for 2019. Additionally, the most recent life-cycle sustainment cost estimate for the F-35 fleet is more than $1 trillion, which DOD officials have deemed unaffordable. The program's long term sustainment estimates reflect assumptions about key cost drivers that the program does not control, including fuel costs, labor costs, and inflation rates. The program is also focusing on product reliability, which is something that the program can control, and something we have found in our prior best practices work to be a key to driving down sustainment costs. According to program reliability data, each F-35 variant was tracking closely to its reliability plan as of December 2013, although the program has a long way to go to achieve its reliability goals. Long-term Acquisition Funding Projections Pose Affordability Challenges: The overall affordability of the F-35 acquisition program remains a significant concern. As of March 2013, the program office estimated that the total acquisition cost will be $390.4 billion. DOD's estimated annual funding levels to finish development and procurement of the F-35 are shown in figure 2. Figure 2: Budgeted Development and Procurement Costs by Service, 2014- 2037 (dollars in then year billions): [Refer to PDF for image: vertical bar graph] Average: $12.6 billion. Fiscal year: 2014; Air Force Development: $0.8; Air Force Procurement: $3.7; Navy Development: $1.0; Navy Procurement: $2.9. Fiscal year: 2015; Air Force Development: $0.6; Air Force Procurement: $4.9; Navy Development: $1.1; Navy Procurement: $3.1. Fiscal year: 2016; Air Force Development: $0.5; Air Force Procurement: $6.2; Navy Development: $0.9; Navy Procurement: $3.8. Fiscal year: 2017; Air Force Development: $0.2; Air Force Procurement: $6.3; Navy Development: $0.5; Navy Procurement: $5.1. Fiscal year: 2018; Air Force Development: $0.08; Air Force Procurement: $7.0; Navy Development: $0.13; Navy Procurement: $6.1. Fiscal year: 2019; Air Force Procurement: $6.6; Navy Procurement: $5.5. Fiscal year: 2020; Air Force Procurement: $6.7; Navy Procurement: $5.8. Fiscal year: 2021; Air Force Procurement: $8.5; Navy Procurement: $5.9. Fiscal year: 2022; Air Force Procurement: $8.5; Navy Procurement: $5.6. Fiscal year: 2023; Air Force Procurement: $8.8; Navy Procurement: $5.7. Fiscal year: 2024; Air Force Procurement: $8.9; Navy Procurement: $5.6. Fiscal year: 2025; Air Force Procurement: $9.0; Navy Procurement: $5.7. Fiscal year: 2026; Air Force Procurement: $9.3; Navy Procurement: $5.6. Fiscal year: 2027; Air Force Procurement: $9.5; Navy Procurement: $5.4. Fiscal year: 2028; Air Force Procurement: $9.4; Navy Procurement: $5.4. Fiscal year: 2029; Air Force Procurement: $9.7; Navy Procurement: $5.4. Fiscal year: 2030; Air Force Procurement: $9.9; Navy Procurement: $5.2. Fiscal year: 2031; Air Force Procurement: $10.3; Navy Procurement: $2.7. Fiscal year: 2032; Air Force Procurement: $10.1; Navy Procurement: $0.7. Fiscal year: 2033; Air Force Procurement: $10.3 Fiscal year: 2034; Air Force Procurement: $10.3. Fiscal year: 2035; Air Force Procurement: $10.5. Fiscal year: 2036; Air Force Procurement: $10.7. Fiscal year: 2037; Air Force Procurement: $9.87. Source: GAO analysis of DOD data. [End of figure] From fiscal years 2014 to 2018, DOD plans to increase development and procurement funding for the F-35 from around $8 billion to around $13 billion, an investment of more than $50 billion over that 5-year period. This build-up will occur during years of potential reductions in DOD's budget as a result of sequestration. From fiscal year 2014 through fiscal year 2037, the program projects that it will require, on average, development and procurement funding of $12.6 billion per year, with several peak years at around $15 billion. Such a high average annual cost requirement poses affordability risks. At $12.6 billion a year, the F-35 acquisition program alone would consume around one-quarter of all of DOD's annual major defense acquisition funding. Therefore, any change in F-35 funding is likely to affect DOD's ability to fully fund its other major acquisition programs. In addition, maintaining this level of sustained funding will be difficult in a period of declining or flat defense budgets and competition with other large acquisition programs such as the KC-46 tanker and a new bomber. These costs do not include the costs to operate and maintain the F-35s as they are produced and fielded. Recognizing the affordability challenges posed by the F-35 program, the Under Secretary of Defense for Acquisition, Technology, and Logistics established affordability unit cost targets for each F-35 variant to be met by the start of full rate production in 2019. The program is likely to be challenged to meet those targets, as the three variants still require anywhere from $41 million to $49 million in unit cost reductions (see table 2). In addition, the program's current funding and quantity projections indicate that unit costs in 2019 could actually be higher than the targets. The Under Secretary issued a memorandum in April 2013 explaining that affordability constraints are intended to force prioritization of requirements, drive performance and cost trades, and ensure that unaffordable programs do not enter the acquisition process. The memorandum goes on to state that "if affordability caps are breached, costs must be reduced or else program cancellation can be expected." Table 2: Comparison of 2013 Unit Costs and 2019 Unit Cost Goals (dollars in then year millions): F-35 variant: Conventional takeoff and vertical landing; Unit cost of aircraft procured in 2013: $124.8; Unit cost target of aircraft to be procured in 2019[A]: $83.4; Amount currently over cost targets: $41.4. F-35 variant: Short takeoff and vertical landing; Unit cost of aircraft procured in 2013: $156.8; Unit cost target of aircraft to be procured in 2019[A]: $108.1; Amount currently over cost targets: $48.7. F-35 variant: Carrier-suitable variant; Unit cost of aircraft procured in 2013: $142.6; Unit cost target of aircraft to be procured in 2019[A]: $93.3; Amount currently over cost targets: $49.3. Source: GAO analysis of DOD data. [A] These targets are based upon planning assumptions reflected in the FY 2013 President's Budget and the 2011 projection for international partner procurement. If there are subsequent changes to either the U.S. or international partner procurement quantities the program will isolate the effect that this has on the targets as a factor that is not within their control. [End of table] The F-35 program made progress this year in decreasing the unit costs of the conventional take-off and landing and carrier-suitable variants, but the unit cost of the short takeoff and vertical landing variant increased by nearly $10 million. According to program officials, the unit cost of the short takeoff and vertical landing variant increased because the program had to delay the procurement of a number of aircraft into the future, which reduced near-term quantities and made each individual unit more costly, and engine costs were higher than originally estimated. There is still uncertainty surrounding these estimates depending upon how DOD chooses to implement sequestration in future budgets. Estimated Sustainment Costs Remain High but Emphasis on Improving Reliability Could Reduce Costs: In addition to the concerns about the affordability of the F-35 acquisition program, there are also significant concerns about the cost of operating and supporting the F-35 fleet over the coming decades. Currently, the Cost Assessment and Program Evaluation (CAPE) office, within the Office of the Secretary of Defense, estimates that the cost to operate and support the fleet over 30 years is likely to exceed $1 trillion, which is 3 times higher than what was projected when the development program began in 2001. CAPE's estimates also indicate that F-35 operations and support costs could surpass the average cost of legacy aircraft by 40 percent or more, when original estimates indicated that the F-35 would cost less than the legacy aircraft. Program officials recently stated that their estimates indicate that operation and support costs are likely to be closer to $860 billion, and not the $1 trillion estimated by CAPE. According to CAPE, program, and contractor officials, F-35 sustainment cost estimates differ as the assumed future values for key cost drivers, like inflation rates and fuel costs, vary among cost estimators. CAPE officials emphasize that the difference between cost estimates is almost entirely attributable to the use of different inflation indices. Table 3 below lists the top cost drivers in the F-35 operation and sustainment estimates. While it is important for the program to consider potential reductions or increases in these variables listed below as it estimates the F-35's long-term operation and sustainment costs, some of those variables can be directly controlled by the program office while others like inflation rates and fuel costs cannot. Table 3: Cost Drivers That Affect F-35 Operation and Sustainment Cost Estimates: Input: On-site repairs and number of spares; Cost driver: Cost of providing maintenance with spares on site; Risk/Uncertainty: Services are in the process of determining the location of F-35 repair work as well as the number of spare parts needed. The services decide locations, but the number of spares may be determined based on other factors outside of military control. Input: Fuel costs; Cost driver: Cost of fuel to fly planes at expected hours; Risk/Uncertainty: Fuel usage is currently based on data gathered from developmental test aircraft. Jet fuel costs are volatile. The cost of jet fuel grew 769 percent for American private flight carriers between 1977 and 2013. Although the services cannot control the cost directly, they can determine the number of hours flown which affects the amount of fuel used. Input: Mission personnel; Cost driver: Cost of employing and training personnel; Risk/Uncertainty: The services have not yet defined the number of personnel needed. Input: Government versus contractor support labor rates; Cost driver: Percentage of work conducted by government versus contractor personnel; cost of reimbursement; Risk/Uncertainty: The F-35 program office has not yet made a decision on the percentage of work to be conducted by government versus contractor personnel. Input: Number and size of squadrons; Cost driver: Includes the number/size of squadrons, and squadron personnel estimates; Risk/Uncertainty: The military services are currently considering the number and size of squadrons. Input: Equipment costs and reliability; Cost driver: Costs of equipment needed to keep fleets operational; Risk/Uncertainty: Development is ongoing, so actual aircraft reliability is not yet known. The services have little control over equipment costs, but can impact reliability as the system is developed. Input: Number of planes and flight hours; Cost driver: Air Force: 1,763 planes at 250 hours; Navy: 340 planes at 316 hours; Marine Corps: 340 planes at 302 hours; Risk/Uncertainty: The services have decreased their anticipated flight hours throughout development, and may do so again in the future. Input: Inflation rates; Cost driver: Annual Inflation indices used by the Department of Defense; Risk/Uncertainty: These rates are adjusted by the Office of Management and Budget annually. Source: GAO Analysis of DOD, contractor, and aircraft industry data. [End of table] The F-35 program office and prime contractor are working to make the long-term program more affordable. Starting in September 2013, they established a sustainment cost initiative team to meet regularly and discuss options for driving down sustainment costs. According to contracting officials, they also developed a management team dedicated to improving the aircraft's prognostics and health management system. Additionally, the program is awaiting the results of a business case analysis of the costs and benefits of various sustainment options. The first phase of that analysis, completed in 2012, found that relying on government personnel for sustainment processes would be less costly than having contractors do the work. The second phase, which began in 2013, is examining multiple aspects of the F-35 sustainment strategy, including the program's approach to maintenance management and supply chain support. The findings of this second phase are expected to identify affordability opportunities and areas for additional future analysis. Those findings are expected to be provided to the program by March 2014. As the program faces key decisions about its F-35 operation and support strategy reliability is still a significant concern. Our past work has found that weapon system operating and support costs are directly correlated to weapon system reliability, which is something the program can affect.[Footnote 16] We found that lower reliability causes an imbalance in the relationship between readiness and operating costs, and lends toward the need for high costs to maintain readiness, as seen in figure 3 below. We also previously found that reliability problems identified in DOD weapon systems resulted in cost overruns and schedule delays.[Footnote 17] Figure 3: Relationship among Readiness, Reliability, and Sustainment Costs: [Refer to PDF for image: illustration] Readiness requirement: Inverse relationship between: Operating and support costs and Reliability rate. Poor outcome: Achievement of low reliability rates during development requires higher operating and support costs once fielded. Best practice outcome: Achievement of high reliability rates during development allows lower operating and support costs once fielded. Source: GAO. [End of figure] DOD and the contractor use various measures to track and improve F-35 reliability, including average flying hours between failures, which is defined as the number of flying hours achieved divided by the number of failures incurred. As indicated in figure 4, the conventional takeoff and landing variant and the short takeoff and vertical landing variant were not meeting expected reliability as of September 2013, while the carrier-suitable variant was performing better than expected. Figure 4: F-35 Average Flying Hours between Failure Rates by Variant: [Refer to PDF for image: vertical bar graph] Conventional take-off and landing: Demonstrated as of September 2013: 3.37 hours; Planned by September 2013: 3.72 hours; Goal at maturity: 6 hours. Short take-off vertical landing: Demonstrated as of September 2013: 2.35 hours; Planned by September 2013: 2.41 hours; Goal at maturity: 4 hours. Carrier variant: Demonstrated as of September 2013: 2.26 hours; Planned by September 2013: 2.06 hours; Goal at maturity: 4 hours. Source: GAO analysis of DOD data. [End of figure] DOT&E's recent report noted concerns about the program's ability to achieve its reliability goals by the time each of the F-35 variants reaches maturity--defined as 75,000 flight hours for the CTOL and STOVL variants; and 50,000 flight hours for the CV. DOT&E also noted that the F-35 design is becoming more stable, and although the program still has a large number of flight hours to go until system maturity, additional reliability growth is not likely to occur without a focused, aggressive and well-resourced effort. F-35 Aircraft Manufacturing Efficiency Continues to Improve: F-35 manufacturing has improved and the contractor's management of its suppliers is evolving. As the number of aircraft in production has increased, learning has taken place and manufacturing efficiency has improved. For example, the prime contractor has seen reductions in overall labor hours needed to manufacture the aircraft. The number of F-35 aircraft produced and delivered annually by the prime contractor has steadily increased since the first low rate production aircraft were delivered in 2011. In 2013, the contractor delivered 35 aircraft to the government, 5 more than it delivered in 2012 and 26 more than it delivered in 2011. The prime contractor has put in place a supplier management system to oversee key supplier performance allowing them to identify poor performers and take appropriate action to address issues such as part shortages and poor quality. According to contractor officials, actions taken as a result of this system contributed to improvements in supplier performance over the past year. Program Manufacturing Processes Continue to Show Improvement: The prime contractor continues to gain efficiencies in the manufacturing process as it learns more about manufacturing the aircraft. Reductions in the amount of time spent on work completed out of its specific work station have contributed to overall labor hour reductions. Aircraft delivered in 2012 averaged about 93 hours of out of station work per aircraft, while in 2013 about 8 hours of out of station work were expended per aircraft on average. While these gains in efficiency have moved the program closer to meeting its established labor hour goals, there is still a long way to go. In 2013, the prime contractor was unable to reach labor hour goals for both the CTOL and STOVL variants.[Footnote 18] By the end of 2014, the prime contractor expects to significantly reduce the average labor hours to produce the CTOL and STOVL variants. However, in order to meet its goal, the program will have to reduce the average number of hours per aircraft for CTOL production by nearly 20,000 and the average number of hours per aircraft for STOVL production by more than 14,000 over the next year. While challenging, this goal appears to be achievable given that the program has been able to reduce the average labor hours for CTOL and STOVL production by more than 20,000 hours annually since 2011. Figure 5 identifies the prime contractor's trend in reduction of labor hours since the beginning of low-rate initial production as well as the contractor's plans for 2014. Figure 5: Average Labor Hours Per Aircraft Since Beginning Low-Rate Production and 2014 Goal: [Refer to PDF for image: vertical bar graph] Year: 2011; Conventional take-off and landing: 124,828; Short take-off and landing: 134,423. Year: 2012; Conventional take-off and landing: 96,210; Short take-off and landing: 105,476. Year: 2013; Conventional take-off and landing: 78,198; Short take-off and landing: 79,562. Year: 2014 Goal; Conventional take-off and landing: 57,827; Short take-off and landing: 65,463. Source: GAO analysis of DOD data. [End of figure] As manufacturing efficiency has improved, the prime contractor has also been able to increase throughput, delivering more aircraft year over year--9 in 2011, 30 in 2012, and 35 in 2013. Over the past year, the prime contractor continued to deliver aircraft closer to contracted delivery dates. Last year we found that deliveries averaged 11 months late, but had improved considerably.[Footnote 19] We found similar results this year with deliveries in 2013 averaging about 5 months late. While deliveries in 2013 were later than planned, the trend continued to move in the right direction. Figure 6 tracks the actual delivery dates against the dates specified in the contract. Figure 6: Trend of F-35 Actual Delivery Dates Compared to Contractual Delivery Dates: [Refer to PDF for image: multiple line graph] Graph depicts the following: Actual delivery date: Contractual delivery date: for each aircraft. Source: GAO analysis of DOD data. [End of figure] The improvements in throughput in 2013 were achieved despite setbacks that kept the contractor from delivering aircraft for several months during the year. Those setbacks included a fleet grounding due to engine problems discovered in testing and the need to repave the runway used by the prime contractor. The prime contractor took actions that helped mitigate the impacts of these events and as a result delivered 35 aircraft in 2013--one aircraft short of its plan. Prime Contractor Supplier Management System: The prime contractor is responsible for managing a complex supply chain made up of a large number of national and international suppliers. Currently, the prime contractor oversees about 1,500 domestic suppliers and 80 international suppliers spread across 11 countries. Figure 7 identifies those countries participating in F-35 production. The number of suppliers has grown significantly over the past three years. Since 2011, 153 new suppliers have been added to the supply chain, 67 of those were added between 2012 and 2013. The prime contractor expects more suppliers to be added as production increases and the program progresses. Figure 7: Countries Currently Participating in F-35 Production: [Refer to PDF for image: world map] Australia; Canada; Denmark; Israel; Italy; Japan; Netherlands; Norway; Turkey; United Kingdom; United States. Source: GAO presentation of DOD data; Map Resources (map). [End of figure] The prime contractor's insight into the performance of 52 of its key suppliers through its Supplier Integration Management system has led to actions that have improved supplier performance. The system tracks supplier performance data in 23 areas including, but not limited to, cost growth, parts shortage occurrences, and the number of corrective action reports filed. That data is reviewed and scored on a monthly basis, with each supplier receiving an overall score based on their performance. According to the prime contractor, a score of 80 or above, out of 100, is considered good performance. All 52 of the key suppliers tracked using this system were considered good performers as of December 2013. In addition, 15 showed improvement in performance over the last year. According to contractor officials, the system identifies poor performers who are counseled and corrective actions are identified and implemented. For example, according to officials from the prime contractor, one supplier was identified through this process as having a large number of parts that did not conform to specifications. The prime contractor held meetings with and provided direction to that supplier. As a result of the prime contractor's actions, the supplier's performance has improved over the last year. In addition, officials from the prime contractor have identified part shortages--parts that are late to production need--as a major concern. The number of part shortages has slightly increased over the last year. Although the root cause of the shortages is still being assessed, officials from the prime contractor stated they are currently working on ways to improve part availability. Conclusions: Since the F-35 program restructuring was completed in March 2012, acquisition cost and schedule estimates have remained relatively stable, and the program has made progress in key areas. However, persistent software problems have slowed progress in mission systems flight testing, which is critical to delivering the warfighting capabilities expected by the military services. These persistent delays put the program's development cost and schedule at risk. As a result, DOT&E now projects that the warfighting capabilities expected by the Marine Corps in July 2015, will not likely be delivered on time, and could be delayed as much as 13 months. This means that the Marine Corps may initially receive less capable aircraft than it expects, and if progress in mission systems software testing continues to be slower than planned, Air Force and Navy initial operational capabilities may also be affected. The program may also have to extend its overall developmental flight test schedule, which would increase concurrency between testing and production and could result in additional development cost growth. In addition to software concerns, the current funding plans may be unaffordable, given current budget constraints. This situation could worsen if unit cost targets are not met. Finally, the estimated cost of operating and supporting the fleet over its life-cycle continues to be high and could increase further if aircraft reliability goals are not met. DOD has already made a number of difficult decisions to put the F-35 on a more sound footing. More such decisions may lie ahead. For example, if software testing continues to be delayed, if funding falls short of expectations, or if unit cost targets cannot be met, DOD may have to make decisions about whether to proceed with production as planned with less capable aircraft or to alter the production rate. Also, if reliability falls short of goals, DOD may have to make decisions about other ways to reduce sustainment costs, such as reduced flying hours. Eventually, DOD will have to make contingency plans for these and other issues. At this point, we believe the most pressing issue is the effect software testing delays are likely to have on the capabilities of the initial operational aircraft that each military service will receive. In order to make informed decisions about weapon system investments and future force structure, it is important that Congress and the services have a clear understanding of the capabilities that the initial operational F-35 aircraft will possess. Recommendation for Executive Action: Due to the uncertainty surrounding the delivery of F-35 software capabilities, we recommend that the Secretary of Defense conduct an assessment of the specific capabilities that realistically can be delivered and those that will not likely be delivered to each of the services by their established initial operational capability dates. The results of this assessment should be shared with Congress and the military services as soon as possible but no later than July 2015. Agency Comments: DOD provided comments on a draft of this report, which are reprinted in appendix III. DOD concurred with our recommendation. We are sending copies of this report to appropriate congressional committees; the Secretary of Defense; the Secretaries of the Air Force, Army, and Navy; the Commandant of the Marine Corps; and the Director of the Office of Management and Budget. The report is available at no charge on the GAO website at [hyperlink, http://www.gao.gov]. If you or your staff have any questions about this report, please contact me at (202) 512-4841 or sullivanm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Staff members making key contributions to this report are listed in appendix IV. Signed by: Michael J. Sullivan: Director: Acquisition and Sourcing Management: List of Committees: The Honorable Carl Levin: Chairman: The Honorable James Inhofe: Ranking Member: Committee on Armed Services: United States Senate: The Honorable Richard Durbin: Chairman: The Honorable Thad Cochran: Ranking Member: Subcommittee on Defense: Committee on Appropriations: United States Senate: The Honorable Howard P. McKeon: Chairman: The Honorable Adam Smith: Ranking Member: Committee on Armed Services: House of Representatives: The Honorable Rodney Frelinghuysen: Chairman: The Honorable Pete Visclosky: Ranking Member: Subcommittee on Defense: Committee on Appropriations: House of Representatives: [End of section] Appendix I: Prior GAO Reports and DOD Responses: 2001: GAO report: GAO-02-39; Estimated development costs: $34.4 Billion; Development Length: 10 years; Aircraft unit cost: $69 Million; Key program event: Start of system development and demonstration approved; Primary GAO message: Critical technologies needed for key aircraft performance elements are not mature. Program should delay start of system development until critical technologies are mature to acceptable levels; DOD response and actions: DOD did not delay start of system development and demonstration stating technologies were at acceptable maturity levels and will manage risks in development. 2005: GAO report: GAO-05-271; Estimated development costs: $44.8 Billion; Development Length: 12 years; Aircraft unit cost: $82 Million; Key program event: The program undergoes re-plan to address higher than expected design weight, which added $7 billion and 18 months to development schedule; Primary GAO message: We recommend that the program reduce risks and establish executable business case that is knowledge-based with an evolutionary acquisition strategy; DOD response and actions: DOD partially concurred but does not adjust strategy, believing that their approach is balanced between cost, schedule and technical risk. 2006: GAO report: GAO-06-356; Estimated development costs: $45.7 Billion; Development Length: 12 years; Aircraft unit cost: $86 Million; Key program event: Program sets in motion plan to enter production in 2007 shortly after first flight of the non-production representative aircraft; Primary GAO message: The program plans to enter production with less than 1 percent of testing complete. We recommend program delay investing in production until flight testing shows that JSF performs as expected; DOD response and actions: DOD partially concurred but did not delay start of production because they believe the risk level was appropriate. 2007: GAO report: GAO-07-360; Estimated development costs: $44.5 Billion; Development Length: 12 years; Aircraft unit cost: $104 Million; Key program event: Congress reduced funding for first two low-rate production buys thereby slowing the ramp up of production; Primary GAO message: Progress is being made but concerns remained about undue overlap in testing and production. We recommend limits to annual production quantities to 24 a year until flying quantities are demonstrated; DOD response and actions: DOD non-concurred and felt that the program had an acceptable level of concurrency and an appropriate acquisition strategy. 2008: GAO report: GAO-08-388; Estimated development costs: $44.2 Billion; Development Length: 12 years; Aircraft unit cost: $104 Million; Key program event: DOD implemented a Mid-Course Risk Reduction Plan to replenish management reserves from about $400 million to about $1 billion by reducing test resources; Primary GAO message: We believe new plan increased risks and DOD should revise it to address concerns about testing, management reserves, and manufacturing concerns. We determined that the cost estimate was not reliable and that a new cost estimate and schedule risk assessment is needed; DOD response and actions: DOD did not revise risk plan or restore testing resources, stating that they will monitor the new plan and adjust it if necessary. Consistent with a report recommendation, a new cost estimate was prepared, but DOD did not conduct a risk and uncertainty analysis. 2009: GAO report: GAO-09-303; Estimated development costs: $44.4 Billion; Development Length: 13 years; Aircraft unit cost: $104 Million; Key program event: The program increased the cost estimate and adds a year to development but accelerated the production ramp up. Independent DOD cost estimate (JET I) projects even higher costs and further delays; Primary GAO message: Moving forward with an accelerated procurement plan and use of cost reimbursement contracts is very risky. We recommended the program report on the risks and mitigation strategy for this approach; DOD response and actions: DOD agreed to report its contracting strategy and plans to Congress and conduct a schedule risk analysis. The program reported completing the first schedule risk assessment with plans to update semi-annually. The Department announced a major program reducing procurement and moving to fixed-price contracts. 2010: GAO report: GAO-10-382; Estimated development costs: $49.3 Billion; Development Length: 15 years; Aircraft unit cost: $112 Million; Key program event: The program was restructured to reflect findings of recent independent cost team (JET II) and independent manufacturing review team. As a result, development funds increased, test aircraft were added, the schedule was extended, and the early production rate decreased; Primary GAO message: Costs and schedule delays inhibit the program's ability to meet needs on time. We recommend the program complete a full comprehensive cost estimate and assess warfighter and IOC requirements. We suggest that Congress require DOD to tie annual procurement requests to demonstrated progress; DOD response and actions: DOD continued restructuring, increasing test resources and lowering the production rate. Independent review teams evaluated aircraft and engine manufacturing processes. Cost increases later resulted in a Nunn-McCurdy breach. Military services are currently reviewing capability requirements as we recommended. 2011: GAO report: GAO-11-325; Estimated development costs: $51.8 Billion; Development Length: 16 years; Aircraft unit cost: $133 Million; Key program event: Restructuring continued with additional development cost increases; schedule growth; further reduction in near-term procurement quantities; and decreased the rate for future production. The Secretary of Defense placed the STOVL variant on a two-year probation; decoupled STOVL from the other variants; and reduced STOVL production plans for fiscal years 2011 to 2013; Primary GAO message: The restructuring actions are positive and if implemented properly, should lead to more achievable and predictable outcomes. Concurrency of development, test, and production is substantial and provides risk to the program. We recommended the program maintain funding levels as budgeted; establish criteria for STOVL probation; and conduct an independent review of software development, integration, and test processes; DOD response and actions: DOD concurred with all three of the recommendations. DOD lifted STOVL probation citing improved performance. Subsequently, DOD further reduced procurement quantities, decreasing funding requirements through 2016. The initial independent software assessment began and ongoing reviews are planned to continue through 2012. 2012: GAO report: GAO-12-437; Estimated development costs: $55.2 Billion; Development Length: 18 years; Aircraft unit cost: $137 Million; Key program event: The program established a new acquisition program baseline and approved the continuation of system development, increasing costs for development and procurements and extending the period of planned procurements by 2 years; Primary GAO message: Extensive restructuring places the program on a more achievable course. Most of the program's instability continues to be concurrency of development, test, and production. We recommend the Cost Assessment Program Evaluation office conduct an analysis on the impact of lower annual funding levels; JSF program office conducts an assessment of the supply chain and transportation network; DOD response and actions: DOD partially concurred with conducting an analysis on the impact of lower annual funding levels and concurred with assessing the supply chain and transportation network. 2013: GAO report: GAO-13-309; Estimated development costs: $55.2 Billion; Development Length: 18 years; Aircraft unit cost: $137 Million; Key program event: The program continued to move forward following a new acquisition program baseline in 2012. In doing so, the program incorporated positive and more realistic restructuring actions taken since 2010 including more time and funding for development and deferred procurement of more than 400 aircraft to future years; Primary GAO message: The program is moving in the right direction but must fully validate design and operational performance and at the same time make the system affordable. We did not make recommendations to DOD; DOD response and actions: DOD agreed with GAO's observations. Source: GAO. [End of table] [End of section] Appendix II: Scope and Methodology: To assess the program's ongoing development and testing we reviewed the status of software development and integration and contractor management improvement initiatives. We also interviewed officials from the program office, prime contractor, and the Defense Contract Management Agency (DCMA) to discuss current development status and software releases. In addition, we compared management objectives to progress made on these objectives during the year. We obtained and analyzed data on flights and test points, both planned and accomplished during 2013. We compared test progress against the total program plans to complete. We also reviewed the Director, Operational Test and Evaluation's annual F-35 assessment. In addition, we interviewed officials from the F-35 program office and aircraft prime contractor to discuss development test plans and achievements. We also collected information from the program office, prime contractor and Department of Defense (DOD) test pilots regarding the program's technical risks including the helmet mounted display, autonomic logistics information system, carrier arresting hook, and structural durability. To assess the program's funding and long-term affordability, we reviewed financial management reports, annual Selected Acquisition Reports, and monthly status reports available as of December 2013. In addition, we reviewed total program funding requirements from the Defense Acquisition Executive Summary. We used this data to project annual funding requirements through the expected end of the F-35 acquisition in 2037. We also analyzed the fiscal year 2014 President's Budget data to identify the current status of unit costs for each variant, and the differences in this cost since 2012. We reviewed the Office of the Secretary of Defense's F-35 Joint Strike Fighter Concurrency Quick Look Review, and discussed and analyzed reported concurrency costs with the prime contractor and program office. We obtained and discussed the life-cycle operating and support cost through the program's Selected Acquisition Report and projections made by the Cost Analysis and Program Evaluation (CAPE) office. We identified changes in cost and interviewed officials from the program office prime contractor, Naval Air Systems Command, and the CAPE office regarding reasons for these changes. We also discussed future plans of the DOD and prime contractor to try and reduce life-cycle sustainment costs with officials from the prime contractor, program office, and CAPE. We analyzed reliability data and discussed these issues with program and prime contractor officials. To assess manufacturing progress we obtained and analyzed data related to aircraft delivery rates and work performance data through the end of calendar year 2013. This data was compared to program objectives identified in these areas and used to identify trends. We reviewed data and briefings provided by the program office, prime contractor, and DCMA in order to identify issues in manufacturing processes. We discussed reasons for delivery delays and plans for improvement with the prime contractor. We also toured the prime contractor's manufacturing facility in Fort Worth, Texas and collected and analyzed data related to aircraft quality through December 2013. We reviewed and discussed information on the prime contractor's global supply chain including their management processes for oversight. We assessed the reliability of DOD and contractor data by reviewing existing information about the data, and interviewing agency officials knowledgeable about the data. We determined that the data were sufficiently reliable for the purposes of this report. We conducted this performance audit from June 2013 to March 2014 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. [End of section] Appendix III: Comments from the Department of Defense: ASSISTANT SECRETARY OF DEFENSE: ACQUISITION 3015 DEFENSE PENTAGON: WASHINGTON, DC 20301-3015: March 19, 2014: Mr. Michael J. Sullivan: Director, Acquisition and Sourcing Management: U.S. Government Accountability Office: 441 G Street, NW: Washington, DC 20548: Dear Mr. Sullivan: This is the Department of Defense (DoD) response to the GAO draft report, GAO-14-322, "F-35 Joint Strike Fighter: Problems Completing Software Testing May Hinder Delivery of Expected Warfighting Capabilities," dated February 28,2014 (GAO Code 121146). Detailed comments on the report are enclosed. The DoD concurs with your recommendation and we appreciate the opportunity to comment on the draft report. My point of contact for this effort is Colonel Todd Levine, 703-697-2573, todd.j.levine.mil@mail.mil. Signed by: Katrina McFarland: Enclosure: As stated: GAO Draft Report Dated February 28, 2014: GAO-14-322 (GAO Code 121146): "F-35 Joint Strike Fighter: Problems Completing Software Testing May Hinder Delivery Of Expected Warfighting Capabilities" Department Of Defense Comments To The GAO Recommendation: Recommendation: The GAO recommends that the Secretary of Defense conduct an assessment of the specific capabilities that realistically can be delivered and those that will not likely be delivered to each of the services by their established initial operating capability dates. The results of this assessment should be shared with Congress and the military services as soon as possible but no later than July 2015. DOD Response: Concur. The Department agrees that there is value in assessing the risk of delivering required capabilities within the stated initial operational capability windows for each military service in order 10 make timely decisions with respect to future investment and force structure. The Department will ensure that in the process of its periodic reviews and reports, this assessment is conducted and shared with the military services and Congress. [End of section] Appendix IV: GAO Contact and Staff Acknowledgments: GAO Contact: Michael Sullivan (202) 512-4841 sullivanm@gao.gov: Staff Acknowledgments: In addition the contact name above, the following staff members made key contributions to this report: Travis Masters, Assistant Director; Marvin Bonner; Peter Anderson, Megan Porter, Roxanna Sun and Abby Volk. [End of section] Related GAO Products: F-35 Joint Strike Fighter: Restructuring Has Improved the Program, but Affordability Challenges and Other Risks Remain. [hyperlink, http://www.gao.gov/products/GAO-13-690T]. Washington, D.C.: June 19, 2013. F-35 Joint Strike Fighter: Program Has Improved in Some Areas, but Affordability Challenges and Other Risks Remain. [hyperlink, http://www.gao.gov/products/GAO-13-500T]. Washington, D.C.: April 17, 2013. F-35 Joint Strike Fighter: Current Outlook Is Improved, but Long-Term Affordability Is a Major Concern. [hyperlink, http://www.gao.gov/products/GAO-13-309]. Washington, D.C.: March 11, 2013. Joint Strike Fighter: DOD Actions Needed to Further Enhance Restructuring and Address Affordability Risks. [hyperlink, http://www.gao.gov/products/GAO-12-437]. Washington, D.C.: June 14, 2012. Joint Strike Fighter: Restructuring Added Resources and Reduced Risk, but Concurrency Is Still a Major Concern. [hyperlink, http://www.gao.gov/products/GAO-12-525T]. Washington, D.C.: March 20, 2012. Joint Strike Fighter: Restructuring Places Program on Firmer Footing, but Progress Is Still Lagging. [hyperlink, http://www.gao.gov/products/GAO-11-677T]. Washington, D.C.: May 19, 2011. Joint Strike Fighter: Restructuring Places Program on Firmer Footing, but Progress Still Lags. [hyperlink, http://www.gao.gov/products/GAO-11-325]. Washington, D.C.: April 7, 2011. Joint Strike Fighter: Restructuring Should Improve Outcomes, but Progress Is Still Lagging Overall. [hyperlink, http://www.gao.gov/products/GAO-11-450T]. Washington, D.C.: March 15, 2011. Tactical Aircraft: Air Force Fighter Force Structure Reports Generally Addressed Congressional Mandates, but Reflected Dated Plans and Guidance, and Limited Analyses. [hyperlink, http://www.gao.gov/products/GAO-11-323R]. Washington, D.C.: February 24, 2011. Joint Strike Fighter: Assessment of DOD's Funding Projection for the F136 Alternate Engine. [hyperlink, http://www.gao.gov/products/GAO-10-1020R]. Washington, D.C.: September 15, 2010. Joint Strike Fighter: Accelerating Procurement before Completing Development Increases the Government's Financial Risk. [hyperlink, http://www.gao.gov/products/GAO-09-303]. Washington D.C.: March 12, 2009. Joint Strike Fighter: Recent Decisions by DOD Add to Program Risks. [hyperlink, http://www.gao.gov/products/GAO-08-388]. Washington, D.C.: March 11, 2008. Joint Strike Fighter: Progress Made and Challenges Remain. [hyperlink, http://www.gao.gov/products/GAO-07-360]. Washington, D.C.: March 15, 2007. Joint Strike Fighter: DOD Plans to Enter Production before Testing Demonstrates Acceptable Performance. [hyperlink, http://www.gao.gov/products/GAO-06-356]. Washington, D.C.: March 15, 2006. Tactical Aircraft: Opportunity to Reduce Risks in the Joint Strike Fighter Program with Different Acquisition Strategy. [hyperlink, http://www.gao.gov/products/GAO-05-271]. Washington, D.C.: March 15, 2005. [End of section] Footnotes: [1] The international partners are the United Kingdom, Italy, the Netherlands, Turkey, Canada, Australia, Denmark, and Norway. These nations contributed funds for system development and signed agreements to procure aircraft. In addition, Israel and Japan have signed on as foreign military sales customers. [2] See related GAO products at the end of this report. [3] See appendix I for a matrix of prior GAO reports, recommendations, and DOD actions. [4] GAO, Joint Strike Fighter: DOD Actions Needed to Further Enhance Restructuring and Address Affordability Risks, [hyperlink, http://www.gao.gov/products/GAO-12-437] (Washington, D.C.: June 14, 2012); and GAO, Joint Strike Fighter: Current Outlook Is Improved, but Long-Term Affordability Is a Major Concern, [hyperlink, http://www.gao.gov/products/GAO-13-309] (Washington, D.C.: Mar. 11, 2013). [5] Pub. L. No. 111-84, § 244 (2009). [6] [hyperlink, http://www.gao.gov/products/GAO-13-309]; [hyperlink, http://www.gao.gov/products/GAO-12-437]; GAO, Joint Strike Fighter: Restructuring Places Program on Firmer Footing, but Progress Still Lags, [hyperlink, http://www.gao.gov/products/GAO-11-325] (Washington, D.C.: April 7, 2011). [7] Section 2433 of title 10 of the United States Code, commonly referred to as Nunn-McCurdy, requires DOD to notify Congress whenever a major defense acquisition program's unit cost experiences cost growth that exceeds certain thresholds. This is commonly referred to as a Nunn-McCurdy breach. Significant breaches occur when the program acquisition unit cost or procurement unit cost increases by at least 15 percent over the current baseline estimate or at least 30 percent over the original estimate. For critical breaches, when these unit costs increase at least 25 percent over the current baseline estimate or at least 50 percent over the original, DOD is required to take additional steps, including conducting an in-depth review of the program. Programs with critical breaches must be terminated unless the Secretary of Defense certifies to certain facts related to the program and takes other actions, including restructuring the program. 10 U.S.C. § 2433a. [8] Initial operational capability is obtained when organizations or units have received a specified number of systems and have the ability to employ and maintain those systems. [9] GAO, Tactical Aircraft: Opportunity to Reduce Risk in the Joint Strike Fighter Program with Different Acquisition Strategy, [hyperlink, http://www.gao.gov/products/GAO-05-271] (Washington, D.C.: March 15, 2005). [10] GAO, Joint Strike Fighter: Additional Costs and Delays Risk Not Meeting Warfighter Requirements on Time, [hyperlink, http://www.gao.gov/products/GAO-10-382] (Washington, D.C.: March 19, 2010). [11] Flight test points are specific, quantifiable objectives in flight plans that are needed to verify aircraft design and performance. [12] [hyperlink, http://www.gao.gov/products/GAO-13-309]; [hyperlink, http://www.gao.gov/products/GAO-12-437]. [13] The President ordered a sequestration of budgetary resources on March 1, 2013 in response to the Balanced Budget and Emergency Deficit Control Act of 1985, Pub. L No. 99-177, § 251, as amended by the Budget Control Act of 2011, Pub. L No. 112-25, § 302. [14] Interim Department of Defense Instruction 5000.02 Operation of the Defense Acquisition System, para. (5)(d)(11)(b)(2) (Nov. 25, 2013). [15] [hyperlink, http://www.gao.gov/products/GAO-13-309]; [hyperlink, http://www.gao.gov/products/GAO-12-437]. [16] See GAO, Best Practices: Setting Requirements Differently Could Reduce Weapon Systems' Total Ownership Costs, [hyperlink, http://www.gao.gov/products/GAO-03-57] (Washington, D.C.: Feb. 11, 2003). [17] See GAO, Best Practices: Increased Focus on Requirements and Oversight Needed to Improve DOD's Acquisition Environment Weapon System Quality, [hyperlink, http://www.gao.gov/products/GAO-08-294] (Washington, D.C.: Feb. 1, 2008). [18] The CV recently entered production, limiting the data that were available for trend analysis. As a result it has been excluded in this assessment. [19] [hyperlink, http://www.gao.gov/products/GAO-13-309]. [End of section] GAO's Mission: The Government Accountability Office, the audit, evaluation, and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO's website [hyperlink, http://www.gao.gov]. Each weekday afternoon, GAO posts on its website newly released reports, testimony, and correspondence. To have GAO e-mail you a list of newly posted products, go to [hyperlink, http://www.gao.gov] and select "E-mail Updates." Order by Phone: The price of each GAO publication reflects GAO's actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO's website, [hyperlink, http://www.gao.gov/ordering.htm]. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. Connect with GAO: Connect with GAO on facebook, flickr, twitter, and YouTube. Subscribe to our RSS Feeds or E mail Updates. Listen to our Podcasts. Visit GAO on the web at [hyperlink, http://www.gao.gov]. To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Website: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]; E-mail: fraudnet@gao.gov; Automated answering system: (800) 424-5454 or (202) 512-7470. Congressional Relations: Katherine Siggerud, Managing Director, siggerudk@gao.gov: (202) 512-4400: U.S. Government Accountability Office: 441 G Street NW, Room 7125: Washington, DC 20548. Public Affairs: Chuck Young, Managing Director, youngc1@gao.gov: (202) 512-4800: U.S. Government Accountability Office: 441 G Street NW, Room 7149: Washington, DC 20548. [End of document]