Recommendation: The Secretary of Defense should ensure the Under Secretary of Defense for Acquisition and Sustainment, in consultation with the F-35 Program Executive Officer, develops a program-wide process for measuring, collecting, and tracking information on how ALIS is affecting the performance of the F-35 fleet to include, but not be limited to, its effects on mission capability rates. (Recommendation 1)

Agency: Department of Defense: Office of the Secretary of Defense
Status: Open

Comments: When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.

Recommendation: The Secretary of Defense should ensure the Under Secretary of Defense for Acquisition and Sustainment, in consultation with the F-35 Program Executive Officer, develops and implements a strategy for the re-design of ALIS. The strategy should be detailed enough to clearly identify and assess the goals, key risks or uncertainties, and costs of re-designing the system. (Recommendation 2)

Agency: Department of Defense: Office of the Secretary of Defense
Status: Open

Comments: When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.

Recommendation: The F-35 Program Executive Officer should formally share or make available, through a new or existing communications mechanism, F-35 operational lessons learned across the services. (Recommendation 1)

Agency: Department of Defense
Status: Open

Comments: DOD concurred with our recommendation. According to DOD officials, the Air Force, the Marine Corps and the Navy all have robust systems for capturing and sharing F-35 operational lessons learned. However, although these systems are accessible by members of the other services, there is a general lack of awareness of how to access systems across military services. As of December 2019, DOD officials stated that they were developing a Lessons Learned Database, which they estimate will be completed during the third quarter of 2020. We are encouraged that the department is aware of the importance of sharing operational lessons learned across the services and that a solution is likely on the horizon. However, until the department implements the solution, this recommendation will remain open.

Recommendation: The Secretary of the Air Force should ensure that the Director of Operations and the Air Force Manpower Analysis Agency reevaluate fighter pilot squadron requirements, to include updating current assumptions of fighter pilot workload, and assessing the impact of future incorporation of UAS platforms into combat aviation. (Recommendation 1)

Agency: Department of Defense: Department of the Air Force
Status: Open

Comments: DOD concurred with this recommendation. In its initial response, DOD noted that across the Air Force, many organizations and offices, in addition to the Director of Operations and Air Force Manpower Analysis Agency will play integral roles in determining the future size and mix of manpower requirements for fighter pilot squadrons. DOD also noted that the Air Force will review fighter wing manpower determinants to accurately account for fighter pilot workloads and analyze support organization manning to ensure adequate support to operational units. In December 2018, the Air Force provided a report to Congress on the efforts to implement GAO's recommendation, among other things. The report noted that an Air Force reevaluation of pilot staff requirements resulted in a reduction to requirements of 106 positions (4 percent). In addition, the report discussed how fighter pilot workload is impacted by UAS platforms, specifically that the growth of UAS platforms has allowed the Air Force to reduce the need for fighter aircraft operations in uncontested environments, and that the growth of the UAS pilot community has allowed that community to begin filling staff positions that would formerly have been filled by fighter pilots. Further, the report stated that the Air Force Manpower Analysis Agency - in coordination with other Air Force offices - was conducting a study of fighter squadron workloads, anticipated to be completed in mid-2019. We requested a copy of the Air For Manpower Analysis Agency study in March 2020, and again in August 2020 - but as of September 2020 a copy has not been provided. Once we have an opportunity to review the study we will update the status of this recommendation as appropriate.

Recommendation: The Secretary of the Navy should ensure that the Chief of Naval Operations reevaluate fighter pilot squadron requirements, to include updating current assumptions of fighter pilot workload, and assessing the impact of future incorporation of UAS platforms into combat aviation. (Recommendation 2)

Agency: Department of Defense: Department of the Navy
Status: Open

Comments: DOD concurred with this recommendation. In its initial response, DOD noted that across the Navy, many organizations and offices including the resource sponsor (Naval Air Forces) will play integral roles in determining the future size and mix of manpower requirements for fighter pilot squadrons. As of August 2020, DOD officials told us that the Navy has updated most of the fighter pilot requirements for most squadrons, is taking action to update requirements for the remaining squadrons, and will provide us with documentation when the process is complete. When we obtain documentation of the updates to fighter pilot squadron requirements we will update the recommendation status as appropriate.

Recommendation: The Secretary of the Navy should ensure that the Commandant of the Marine Corps and the Deputy Commandant for Aviation reevaluate fighter pilot squadron requirements. (Recommendation 3)

Agency: Department of Defense: Department of the Navy
Status: Open

Comments: DOD concurred with this recommendation. In its initial response, DOD noted that across the Marine Corps, many organizations and offices in addition to the Deputy Commandant for Aviation play integral roles in the continuous evaluation and determination regarding current and future size and mix of manpower requirements for fighter and attack squadrons. As of August 2020 DOD has not taken actions in response to this recommendation.

Recommendation: The Under Secretary of Defense for Acquisition, Technology, and Logistics, in coordination with the F-35 Program Executive Officer, should re-examine the metrics that it will use to hold the contractor accountable under the fixed-price, performance-based contracts to ensure that such metrics are objectively measurable, are fully reflective of processes over which the contractor has control, and drive desired behaviors by all stakeholders. (Recommendation 2)

Agency: Department of Defense
Status: Open
Priority recommendation

Comments: DOD concurred with this recommendation. As of December 2019, documentation provided by DOD stated that it had established a joint working group to assess the feasibility of and examine options for DOD's planned 5-year performance-based logistics contract. A DOD official said that this group is also working to identify appropriate metrics to hold the contractor accountable under a potential long-term performance-based contract. DOD has not provided us with a timeline of when the working group's assessment will be complete. We will continue to monitor DOD's efforts to re-examine metrics to ensure that they are objectively measureable, fully reflective of processes over which the contractor has control, and drive desired behaviors by all stakeholders.

Recommendation: The Under Secretary of Defense for Acquisition, Technology, and Logistics, in coordination with the F-35 Program Executive Officer, should, prior to entering into multi-year, fixed-price, performance-based contracts, ensure that DOD has sufficient knowledge of the actual costs of sustainment and technical characteristics of the aircraft after baseline development is complete and the system reaches maturity. (Recommendation 3)

Agency: Department of Defense
Status: Open
Priority recommendation

Comments: DOD concurred with this recommendation. As of December 2019, documentation provided by DOD stated that it had established a joint working group to assess the feasibility of and examine options for DOD's planned 5-year performance-based logistics (PBL) contract. As a product of this assessment, DOD expects that the department will be able to outline what level of knowledge is required of the actual costs of sustainment and technical characteristics of the aircraft in order to enter into a PBL sustainment construct. DOD has not provided us with a timeline of when the working group's assessment will be complete. We will continue to monitor DOD's efforts in this area.

Recommendation: To ensure that annual training plans are aligned with the Air Force's stated goals to ensure that its forces can successfully achieve missions across a broad range of current and emerging threats, the Secretary of Defense should direct the Secretary of the Air Force to comprehensively reassess the assumptions underlying its annual training requirements--including, but not limited to, the total annual training requirements by aircraft, the criteria for designating aircrews as experienced or inexperienced, and the mix between live and simulator training--and make any appropriate adjustments in future training plans.

Agency: Department of Defense
Status: Open

Comments: DOD non-concurred with the recommendation. The Air Force has taken steps to address this recommendation. Specifically, Air Force officials stated they have completed one study and have an ongoing study, intended to reassess the assumptions underlying its annual training requirements for fighter aircrews. For example, Air Force officials stated a study was completed in August 2017 reassessing the criteria for designating aircrews as experienced or inexperienced for 4th generation fighter aircraft. In addition, Air Force officials stated that another study was intended to define the optimum mix of annual training requirements for fighter aircrews. These officials stated that the study results were provided to Air Force senior leaders in July 2018 for approval. As of August 2020, the Air Force did not provide any additional documentation on steps taken to address the recommendation. Completion of these studies and the corresponding adjustments to annual training requirements should help the Air Force ensure that their training plans are aligned to achieve a range of missions for current and emerging threats as recommended by GAO in September 2016.

Recommendation: To improve the Air Force's ability to consistently monitor training results and better position it to allocate resources to address factors that limit the effectiveness of training, the Secretary of Defense should direct the Secretary of the Air Force to establish desired learning objectives and training support elements needed to accomplish the training expectations in its annual Ready Aircrew Program tasking memorandums, and develop a process to collect data to assess the effectiveness of annual training against these features.

Agency: Department of Defense
Status: Open

Comments: DOD non-concurred with the recommendation, stating that the Air Force's Ready Aircrew Program training differs significantly from other syllabus-directed courses of instruction and that desired learning objectives for this training are set at the squadron level in accordance with current Air Force guidance. As of August 2020, DOD did not provide any documentation on steps taken to address this recommendation.

Recommendation: To ensure that risks associated with ALIS are addressed expediently and holistically, the Secretary of Defense should direct the F-35 Program Executive Officer to improve the reliability of its cost estimates, conduct uncertainty and sensitivity analyses consistent with cost-estimating best practices identified in GAO's Cost Estimating and Assessment Guide.

Agency: Department of Defense
Status: Open

Comments: DOD partially concurred with this recommendation. According to DOD officials, as of July 2018, this recommendation conflicts with established Office of the Under Secretary of Defense/Cost Estimation and Program Evaluation guidance for cost estimation and uncertainty analysis. Absent a change in policy at that level, the Joint Program Office will continue to follow Office of the Under Secretary of Defense/Cost Estimation and Program Evaluation policy on this issue. We continue to believe that in order for any risks associated with ALIS to be addressed expediently and holistically, uncertainty and sensitivity analysis must be used on the F-35s cost estimates to improve its overall reliability. Thus, this recommendation will remain open.

Recommendation: To ensure that risks associated with ALIS are addressed expediently and holistically, the Secretary of Defense should direct the F-35 Program Executive Officer to improve the reliability of its cost estimates, ensure that future estimates of ALIS costs use historical data as available and reflect significant program changes consistent with cost-estimating best practices identified in GAO's Cost Estimating and Assessment Guide.

Agency: Department of Defense
Status: Open

Comments: DOD partially concurred with this recommendation. According to DOD officials, since April 2016, the F-35 program has continued to update the ALIS estimate with the latest available cost data, based on recent contracts. Until more reliable actual costs become available, the program utilizes negotiated contract costs, incorporates program initiatives, and ensures the estimate reflects the latest technical baseline and requirements. Until actual costs associated with ALIS historical data are incorporated in the F-35 cost estimate, we believe that the estimate will not be as reliable as it could be. For this reason, this recommendation will remain open.

Recommendation: To help DOD address key risks to F-35 affordability and operational readiness, and to improve the reliability of its O&S cost estimates for the life cycle of the program, the Secretary of Defense should direct the F-35 Program Executive Officer, to enable DOD to better identify, address, and mitigate performance issues with the Autonomic Logistics Information System (ALIS) that could have an effect on affordability, as well as readiness, to establish a performance-measurement process for ALIS that includes, but is not limited to, performance metrics and targets that (1) are based on intended behavior of the system in actual operations and (2) tie system performance to user requirements.

Agency: Department of Defense
Status: Open
Priority recommendation

Comments: DOD concurred with our recommendation. According to DOD officials, as of July 2018, plans are emerging between the services and the Joint Program Office on a path forward for ALIS, focusing on both the current iteration of ALIS and the future state. Going forward, the services and the Joint Program Office are developing plans for the necessary re-architecture of ALIS. Once these current improvements and future requirements are finalized, appropriate performance metrics, tying system performance in operations environments to user requirements, will be incorporated. As of January 2020, DOD officials stated that there was no update to this status. Although DOD has a way ahead as it relates to developing performance metrics for ALIS, DOD has yet to develop any metrics that are based on intended behavior of the system and tie system performance to user requirements. Until DOD takes this action, our recommendation will remain open.

Recommendation: To help DOD address key risks to F-35 affordability and operational readiness, and to improve the reliability of its O&S cost estimates for the life cycle of the program, the Secretary of Defense should direct the F-35 Program Executive Officer, to develop a high level of confidence that the aircraft will achieve its R+M goals, to develop a software reliability and maintainability (R+M) assessment process, with metrics, by which the program can monitor and determine the effect that software issues may have on overall F-35 R+M issues.

Agency: Department of Defense
Status: Open

Comments: DOD concurred with our recommendation. According to DOD officials, as of July 2018, the department and the Joint Program Office, as part of their focus on agile software development, are working to incorporate software reliability and maintainability metrics into future software development and sustainment contracts. Some of the proposed metrics under consideration include: change failure rate; number of errors in developmental/user/operational testing; time to fix on critical errors; and mean time to restore. As of September 2019, DOD officials stated that there was no update to this status. Although attention is being paid to software Reliability & Maintainability, until DOD develops a process focused on software and its effects on overall Reliability & Maintainability issues, this recommendation will remain open.

Recommendation: To help DOD address key risks to F-35 affordability and operational readiness, and to improve the reliability of its O&S cost estimates for the life cycle of the program, the Secretary of Defense should direct the F-35 Program Executive Officer, to promote competition, address affordability, and inform its overarching sustainment strategy, to develop a long-term Intellectual Property (IP) Strategy to include, but not be limited to, the identification of (1) current levels of technical data rights ownership by the federal government and (2) all critical technical data needs and their associated costs.

Agency: Department of Defense
Status: Open
Priority recommendation

Comments: DOD concurred with our recommendation. According to DOD officials, as of January 2020, in the updated F-35 Life Cycle Sustainment Plan (LCSP) issued in January 2019, "Secure Use of Appropriate Technical Data" was one of the identified elements of success necessary to improve F-35 readiness and reduce sustainment costs. As part of the ongoing Plans of Action & Milestones (POAM) implementation process for the LCSP success elements, the F-35 Joint Program Office is working with the OEMs to determine the data rights the government already has, and to determine the specific technical data the government needs, and what it needs that data to accomplish. Significant progress has been made on both fronts with the prime contractor. We acknowledge that progress surrounding technical data rights is being made; however, until an Intellectual Property strategy is developed and released, this recommendation will remain open.