Recommendation: The Commissioner of CBP should ensure that the Biometric Entry-Exit Program's privacy notices contain complete and current information, including all of the locations where facial recognition is used and how travelers can request to opt out as appropriate. (Recommendation 1)

Agency: Department of Homeland Security: United States Customs and Border Protection
Status: Open

Comments: When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.

Recommendation: The Commissioner of CBP should ensure that the Biometric Entry-Exit Program's privacy signage is consistently available at all locations where CBP is using facial recognition. (Recommendation 2)

Agency: Department of Homeland Security: United States Customs and Border Protection
Status: Open

Comments: When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.

Recommendation: The Commissioner of CBP should direct the Biometric Entry-Exit Program to develop and implement a plan to conduct privacy audits of its commercial partners', contractors', and vendors' use of personally identifiable information. (Recommendation 3)

Agency: Department of Homeland Security: United States Customs and Border Protection
Status: Open

Comments: When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.

Recommendation: The Commissioner of CBP should develop and implement a plan to ensure that the biometric air exit capability meets its established photo capture requirement. (Recommendation 4)

Agency: Department of Homeland Security: United States Customs and Border Protection
Status: Open

Comments: When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.

Recommendation: The Commissioner of CBP should develop a process by which Biometric Entry-Exit program officials are alerted when the performance of air exit facial recognition falls below established thresholds. (Recommendation 5)

Agency: Department of Homeland Security: United States Customs and Border Protection
Status: Open

Comments: When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.

Recommendation: The Director, MDA, should ensure an independent assessment is conducted of the agency's process for developing and executing its annual BMDS flight test plan. (Recommendation 1)

Agency: Department of Defense: Office of the Secretary of Defense: Missile Defense Agency
Status: Open

Comments: DOD concurred with the recommendation stating that MDA will conduct an independent assessment as recommended.

Recommendation: The OUSD (A&S) should direct the F-35 program office to provide information that is similar to that which is statutorily required after the milestone C review to Congress ahead of the milestone C review (full-rate production decision). This submission should include an evaluation of the production risks associated with critical production processes that are not in control, reliability and maintainability (R&M) targets that are not met, and supplier readiness—particularly for those replacing Turkish suppliers, along with the steps it is taking to address those risks. (Recommendation 1)

Agency: Department of Defense
Status: Open

Comments: DOD did not concur with this recommendation, citing updates the F-35 program office provides to oversight entities within DOD. We maintain that the program office should provide these same updates to Congress as well. Without a substantive assessment highlighting specific production risks, as well as the steps DOD will take to mitigate them, Congress may not have key insights into the risks that remain with the program and to the overall effort to deliver F-35s to the warfighter. DOD has agreed to keep the Congress apprised of these matters in its quarterly briefings to the defense committees.

Recommendation: The OUSD (A&S) should direct the F-35 program office to establish a Block 4 cost estimate baseline that includes all Block 4 costs, including incurred costs and future costs in its reports to Congress as required by the NDAA for Fiscal Year 2017, so that Congress has a complete understanding of all Block 4 costs and can compare this baseline to future cost estimates and performance. (Recommendation 2)

Agency: Department of Defense
Status: Open

Comments: DOD concurred with this recommendation. In response to our report, DOD stated that future reports to Congress will include prior and future costs, outside of the Future Years Defense Program costs. As of September 15, 2020, DOD has not yet issued an updated report to Congress. We will monitor DOD's efforts to address this recommendation once it releases its next report.

Recommendation: The OUSD (A&S) should direct the F-35 program office to complete a program office level, product-oriented work breakdown structure for the next update to its Block 4 cost estimate to ensure that the estimate meets the comprehensive leading practices. (Recommendation 3)

Agency: Department of Defense
Status: Open

Comments: Though DOD did not concur with this recommendation, in response to this report, DOD agreed to evaluate moving to a program-level, product-oriented work breakdown structure in 2021. We will continue to monitor DOD's efforts to address this recommendation.

Recommendation: The OUSD (A&S) should direct the F-35 program office to conduct risk and uncertainty analyses for the next update to its Block 4 cost estimate to ensure that the estimate meets the credible leading practices. (Recommendation 4)

Agency: Department of Defense
Status: Open

Comments: DOD concurred with this recommendation. In response to this report, DOD stated that the F-35 program office estimate is aligned with the DOD's Cost Assessment and Program Evaluation's approach and methodology for performing development cost estimates. However, DOD did not identify specific actions it plans to take to include risk and uncertainty into its next Block 4 cost estimate update. We will continue to monitor any actions DOD takes in this regard.

Recommendation: The OUSD (A&S) should direct the F-35 program office to consider the results of its future technology readiness assessment of all Block 4 technologies and incorporate the cost and schedule risks of developing those technologies in the next update to its Block 4 cost estimate to ensure that the estimate meets the comprehensive leading practices. (Recommendation 5)

Agency: Department of Defense
Status: Open

Comments: DOD concurred with this recommendation. In response to this report, DOD stated that it continues to evaluate technology readiness levels for Block 4. It noted that Office of the Under Secretary of Defense for Research and Engineering would conduct and independent technology readiness assessment of the Block 4 effort, which the program would use to inform future cost estimates. In May 2020, the F-35 Program Office issued a Technology Readiness Assessment Plan for Block 4. This plan outlines an incremental assessment approach aligned with Block 4 capability drops beginning with the drop scheduled for April 2021. We will continue to monitor DOD's efforts to address this recommendation.

Recommendation: Congress should consider revising Section 224(d) of the National Defense Authorization Act for Fiscal Year 2017, Pub. L. No. 114-328, to extend DOD's Block 4 reporting requirement until all Block 4 capabilities are fielded to ensure that Congress is aware of cost and schedule growth beyond 2023. (Matter for Consideration 1)

Agency: Congress
Status: Open

Comments: Congress has not extended the Block 4 reporting requirement in Section 224(d) of the National Defense Authorization Act for Fiscal Year 2017. We will continue to monitor any action the Congress may take in this regard.

Recommendation: The Secretary of Defense should ensure the Under Secretary of Defense for Acquisition and Sustainment, in consultation with the F-35 Program Executive Officer, develops a program-wide process for measuring, collecting, and tracking information on how ALIS is affecting the performance of the F-35 fleet to include, but not be limited to, its effects on mission capability rates. (Recommendation 1)

Agency: Department of Defense: Office of the Secretary of Defense
Status: Open

Comments: When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.

Recommendation: The Secretary of Defense should ensure the Under Secretary of Defense for Acquisition and Sustainment, in consultation with the F-35 Program Executive Officer, develops and implements a strategy for the re-design of ALIS. The strategy should be detailed enough to clearly identify and assess the goals, key risks or uncertainties, and costs of re-designing the system. (Recommendation 2)

Agency: Department of Defense: Office of the Secretary of Defense
Status: Open

Comments: When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.

Recommendation: The Secretary of Homeland Security should ensure that the Undersecretary for Management develops an oversight process to confirm that programs' schedule goals are developed and updated in accordance with GAO's Schedule Assessment Guide, to include ensuring traceability between APB schedule goals and IMSs. (Recommendation 1)

Agency: Department of Homeland Security: Office of the Secretary
Status: Open

Comments: In providing comments on this report, DHS concurred with our recommendation and stated that the Management Directorate's Office of Program Accountability and Risk Management (PARM) developed a checklist based on GAO's Schedule Assessment Guide, among other things, to evaluate programs' IMSs. PARM also plans to develop guidance on schedules which is intended to assist the Component Acquisition Executives (CAE) and acquisition program staff responsible for building IMSs and APBs. In July 2020, PARM officials reported that the schedule checklist was already being used to evaluate and ensure program IMSs adhered to GAO's Schedule Assessment Guide. In addition, PARM officials are updating and drafting guidance on schedules to assist CAEs and program staff when building IMSs and APBs. As of July 2020, PARM was still in the process of executing these efforts.

Recommendation: The Secretary of Homeland Security should ensure that the Undersecretary for Management revises the schedule development guidance in the Systems Engineering Life Cycle Guidebook to state clearly that an IMS should be used as the basis for APB schedule goals. (Recommendation 2)

Agency: Department of Homeland Security: Office of the Secretary
Status: Open

Comments: DHS concurred with our recommendation and stated that Management Directorate's Office of Program Accountability and Risk Management (PARM) was in the process of revising the Systems Engineering Life Cycle Guidebook and would clarify the language relating to IMSs to ensure guidance is consistent. As of July 2020, PARM was still in the process of revising the Systems Engineering Life Cycle Guidebook.

Recommendation: The Secretary for Homeland Security should direct the Director of OTE to revise T&E policy or guidance as necessary to fully meet the key practice for programs to test that components and subsystems work together as a system in a controlled setting before finalizing a system's design. (Recommendation 1)

Agency: Department of Homeland Security
Status: Open

Comments: In providing comments on this report DHS concurred with our recommendation and stated that it planned to update its T&E policy to specify that acquisition programs demonstrate that components and subsystems work together before finalizing a system's design. In July 2020, DHS Test and Evaluation Division (TED) officials said they were in the process of updating the policy and that it was undergoing management review with an anticipated completion in fall 2020. Once finalized, GAO will evaluate the revised policy to determine whether DHS has met the intent of this recommendation.

Recommendation: The Secretary for Homeland Security should direct the Director of OTE, in coordination with OCPO, to develop an assessment process-including establishing performance measures-to help ensure T&E training achieves desired results. (Recommendation 2)

Agency: Department of Homeland Security
Status: Open

Comments: In providing comments on this report, DHS concurred with our recommendation and stated that it planned to assess the knowledge and skill requirements for the T&E workforce and establish performance goals for the training. DHS Test and Evaluation Division (TED), in coordination with OCPO, also plan to develop strategies to address any deficiencies with the current training that do not meet the identified requirements. In April 2020, TED officials said that they developed a new survey process to obtain recurring feedback from participants on the training's impact on their ability to perform T&E duties as assigned over time to inform the annual review of the T&E curriculum. However, this effort is still in a piloting stage so the extent to which this information is used to assess the training is still unknown at this time. As of July 2020, TED was still in the process of executing these efforts.

Recommendation: The Secretary for Homeland Security should direct the Director of OTE to revise T&E policy or guidance as necessary to specify when in the acquisition life cycle a major acquisition program manager should designate a level III certified T&E manager. (Recommendation 3)

Agency: Department of Homeland Security
Status: Open

Comments: In providing comments on this report DHS concurred with our recommendation and stated that it planned to update its T&E policy to specify when in the acquisition lifecycle acquisition program managers should designate a qualified T&E manager. In July 2020, DHS Test and Evaluation Division (TED) officials said they were in the process of revising the policy to include this specification and that it was undergoing management review with an anticipated completion in fall 2020. Once finalized, GAO will evaluate the revised policy to determine whether DHS has met the intent of this recommendation.

Recommendation: The Secretary for Homeland Security should direct the Director of OTE to establish an internal control process to ensure that data collected and maintained on major acquisition programs' T&E managers are reliable. (Recommendation 4)

Agency: Department of Homeland Security
Status: Open

Comments: In providing comments on this report, DHS concurred with our recommendation and stated that it planned to establish an internal control process to reliably collect and maintain data on acquisition programs' assigned test and evaluation managers. In April 2020, DHS Test and Evaluation Division (TED) reported taking steps to ensure the validity of this data including establishing points of contacts within each component to cross-check collected information for accuracy and having the Director review collected data on a quarterly basis beginning in third quarter fiscal year 2020. As of July 2020, TED was still in the process of improving its internal collection process, but had not completed these efforts.

Recommendation: The Secretary for Homeland Security should direct the Under Secretary for Science and Technology to assess OTE's workforce to ascertain the extent to which it has the appropriate number of staff with the necessary skills to fulfill its responsibilities. (Recommendation 5)

Agency: Department of Homeland Security
Status: Open

Comments: In providing comments on this report, DHS concurred with our recommendation and stated that it planned to assess the Test and Evaluation Division's (TED) workforce by reviewing current staffing levels and vacancies against the division's roles and responsibilities. The Senior Official Performing the Duties of the Under Secretary for Science and Technology plans to use the results of this review to inform strategic hiring in future years, if needed. In February 2020, DHS released its fiscal year 2020 strategic guidance memorandum for the Science and Technology (S&T) Directorate which included a statement pertaining to resourcing S&T's test and evaluation capabilities. However, as of July 2020, S&T had not yet conducted its review of TED's workforce.

Recommendation: The Secretary of Defense should ensure that the KC-46 program office disseminates insights we identified in this report about the KC-46's contracting and sustainment planning experiences for consideration by acquisition programs, in particular those considering a fixed-price-type development contract or a commercial derivative aircraft.

Agency: Department of Defense
Status: Open

Comments: DOD agreed with this recommendation and, as of the summer 2020, has drafted a report that is with the Secretary of the Air Force for review prior to issuance.

Recommendation: The Director, MDA, should utilize additional schedule margin afforded by the EPAA Phase 3 delay to conduct additional testing necessary to thoroughly assess the capabilities and limitations of Phase 3 against IRBMs and a raid scenario prior to delivery. (Recommendation 1)

Agency: Department of Defense: Office of the Secretary of Defense: Missile Defense Agency
Status: Open

Comments: The Department of Defense partially concurred with this recommendation, stating that all EPAA Phase 3 BMDS functions requiring a flight test environment were already successfully demonstrated and that MDA has addressed the intent of our recommendation by adding ground tests to further assess EPAA Phase 3 Capabilities. However, in order for the agency to meet the full intent of our recommendation, additional flight testing to demonstrate capability against EPAA Phase 3 threats, including intermediate-range threats and raid scenarios, is necessary.

Recommendation: The Secretary of Defense should ensure that the F-35 program office assesses whether the Operational Requirements Document (ORD) R&M targets are still feasible and revise the ORD accordingly. (Recommendation 1)

Agency: Department of Defense
Status: Open

Comments: DOD concurred with our recommendation and stated that it would review its Reliability & Maintainability (R&M) requirements and possibly revise them. In late 2019, the F-35 Program Executive Officer (PEO) developed an initiative to clearly identify what, if any, revisions DOD should make to the ORD. As of July 2020, no additional actions have been taken.

Recommendation: The Secretary of Defense should ensure that the F-35 program office, as it revises its Reliability and Maintainability Improvement Program (RMIP), identifies specific and measurable R&M objectives in its RMIP guidance. (Recommendation 2)

Agency: Department of Defense
Status: Open

Comments: DOD concurred with our recommendation and stated that it would update its Reliability & Maintainability (R&M) RMIP guidance. In late 2019, DOD reported that a revised RMIP will be delivered to program leadership for approval. As of July 2020, no additional actions have been taken.

Recommendation: The Secretary of Defense should ensure that the F-35 program office, as it revises its RMIP, identifies and documents which RMIP projects will achieve the identified objectives of the RMIP guidance. (Recommendation 3)

Agency: Department of Defense
Status: Open

Comments: DOD concurred with our recommendation and stated that it would update its RMIP guidance. In late 2019, DOD reported that a revised RMIP will be delivered to program leadership for approval. As of July 2020, no additional actions have been taken.

Recommendation: The Secretary of Defense should ensure that the F-35 program office prioritizes funding for the RMIP. (Recommendation 4)

Agency: Department of Defense
Status: Open

Comments: DOD concurred with our recommendation and stated that it would plan for R&M funding going forward. In Sept. 2019, the F-35 Program Office reported that it was coordinating with the services to increase its Reliability & Maintainability investment. In Nov. 2019, the F-35 Program Office reported that it would allocate significant additional funding for RMIP for calendar year 2020. As of July 2020, no additional actions have been taken.

Recommendation: The Under Secretary for Management should require the Office of Program Accountability and Risk Management to assess the results of major acquisition programs' post-implementation reviews and identify opportunities to improve performance across the acquisition portfolio. (Recommendation 2)

Agency: Department of Homeland Security
Status: Open

Comments: In providing comments on this report, DHS concurred with our recommendation and stated that it planned to update its policy to require more formal reporting requirements and execution criteria for post-implementation reviews. PARM also plans to initiate a study focused on assessing lessons learned and developing a tool to share them across components to improve performance of the acquisition portfolio. In February 2020, PARM approved guidance intended to standardize analysis elements for post-implementation reviews. As of July 2020, PARM was still in the process of developing a tool to share results across components, but in the interim, results from some post-implementation reviews have been shared during meetings with Component Acquisition Executives. GAO will review post-implementation reviews conducted under the new guidance and will monitor DHS's implementation of the tool to ensure the department's actions meet the intent of this recommendation.

Recommendation: The F-35 Program Executive Officer should formally share or make available, through a new or existing communications mechanism, F-35 operational lessons learned across the services. (Recommendation 1)

Agency: Department of Defense
Status: Open

Comments: DOD concurred with our recommendation. According to DOD officials, the Air Force, the Marine Corps and the Navy all have robust systems for capturing and sharing F-35 operational lessons learned. However, although these systems are accessible by members of the other services, there is a general lack of awareness of how to access systems across military services. As of December 2019, DOD officials stated that they were developing a Lessons Learned Database, which they estimate will be completed during the third quarter of 2020. We are encouraged that the department is aware of the importance of sharing operational lessons learned across the services and that a solution is likely on the horizon. However, until the department implements the solution, this recommendation will remain open.

Recommendation: The Secretary of the Navy should take steps to ensure that the Marine Corps not enter full-rate production until the Marine Corps has determined that the contractor has achieved an MRL of at least 9 for all risk areas. (Recommendation 2)

Agency: Department of Defense
Status: Open

Comments: DOD partially concurred with this recommendation, stating that it agreed that manufacturing readiness levels (MRL) for the Amphibious Combat Vehicle 1.1 should be assessed prior to the decision to award the option for the second lot of low-rate production, but disagreed that an MRL assessment of any individual risk area, in itself, should delay the contract award. We maintained that achieving an overall MRL-9 by the by the start of full-rate production represents best practice to minimize production risk. According the Marine Corp, the most recent MRL Assessment as of August 2020 assessed the program at an overall MRL-8. The Marine Corps indicated that another MRL assessment is estimated to be completed in November 2020 to support the decision to enter full-rate production, which is planned for December 2020.

Recommendation: Early in the next decennial cycle, the Secretary of Commerce should direct the Under Secretary of the Economics and Statistics Administration and the Director of the U.S. Census Bureau to plan and execute more flexible, and perhaps smaller, address canvassing test and evaluation activity needed to support key design decisions having significant effect on the cost and quality of the census.

Agency: Department of Commerce
Status: Open

Comments: As of September 2020, the Bureau had not yet begun its 2030 testing and evaluation planning. When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.

Recommendation: The Secretary of Defense should direct the Secretary of the Navy to revise the Navy's ship delivery policy to clarify what types of deficiencies need to be corrected and what mission capability (including the levels of quality and capability) must be achieved at (1) delivery and (2) when the ship is provided to the fleet (at the obligation work limiting date (OWLD)). In doing so, the Navy should clearly define what constitutes a complete ship and when that should be achieved.

Agency: Department of Defense
Status: Open
Priority recommendation

Comments: DOD disagreed with our recommendation to clarify the Navy's ship delivery policy and stated that other existing policies help ensure the completion and capability of ships at delivery. However, as of August 2020, Navy officials stated that they are working on a new response based on ongoing discussions with GAO and they expect to develop a new proposal for responding to this recommendation by December 2020. We maintain that the Navy's ship delivery policy is a key instruction for ensuring that complete, mission-capable ships are provided to the fleet and should be revised in line with our recommendation.

Recommendation: The Secretary of Defense should direct the Secretary of the Navy to reconcile policy with practice to support INSURV's role in making a recommendation for fleet introduction. Accomplishing this may require a study of the current timing of ship trials, and the costs and benefits associated with adding an INSURV assessment prior to providing ships to the fleet.

Agency: Department of Defense
Status: Open

Comments: DOD and the Navy did not concur with this recommendation, noting that the current timing of Navy Board of Inspection and Survey (INSURV) trials provides the Navy with an opportunity to ensure contractual obligations have been met and identify construction deficiencies for correction during the post-delivery period. DOD and the Navy also stated that adding another INSURV trial at the end of the post-delivery period would not be cost-effective and could delay ship deployment schedules. However, we found that most of the significant construction deficiencies identified prior to delivery were not corrected until the post-delivery period and, therefore, INSURV generally did not have an opportunity to inspect these corrections before ships were provided to the fleet. Given this, we maintain that the Navy should re-assess the timing of its post-delivery trials in support of INSURV's responsibility to make recommendations for fleet introduction. As of August 2020, DOD officials stated that they are working on a proposal to address this recommendation based on ongoing discussions with GAO and they expect to complete this proposal by December 2020.

Recommendation: The Secretary of Defense should direct the Secretary of the Navy to reflect additional ship milestones in Selected Acquisition Reports to Congress, including OWLD and readiness to deploy.

Agency: Department of Defense
Status: Open

Comments: DOD partially concurred with this recommendation. DOD and the Navy agreed to report obligation work limiting dates (OWLD) in its Selected Acquisition Reports to Congress, and, as of December 2018, has implemented this portion of the recommendation. The department added the OWLDs for all ships that have yet to achieve this milestone to its Selected Acquisition Reports and plans to continue reporting this information in all subsequent Selected Acquisition Reports. However, DOD did not agree to report ready-to-deploy dates in the Selected Acquisition Reports to Congress, noting that operational factors outside of acquisition concerns can affect the timing of this milestone. While we agree that readiness to deploy is a fleet determination, we continue to believe that this date is important for Congressional oversight, as it remains the best milestone for determining when a ship has achieved a sufficient level of completeness to operate, under the Navy's current framework for ship delivery. As of August 2020, DOD and Navy officials stated that they are working on a new proposal for addressing the recommendation based on ongoing discussions with GAO and they expect to complete this proposal by December 2020.

Recommendation: To strengthen MDA's acquisition efforts and strengthen oversight, and to increase traceability and insight into MDA's test program, the Secretary of Defense should require MDA to (a) include a detailed crosswalk of changes to each test, such as names, planned execution dates, test types, targets, and other modifications, in each iteration of its Integrated Master Test Plan; (b) address deficiencies in its test scheduling policy by better aligning it with best practices for scheduling, including the use of a schedule work breakdown structure (WBS) that clearly traces each activity to the cost WBS, properly assigning resources to schedules, and clarifying guidance on when and how to conduct schedule risk analysis; (c) rectify deficiencies in its element and test level cost estimates by requiring the use of the common test WBS, documenting the traceability of source data, and codifying the processes and associated information for the software application Test Resource Mission Planning-Tool used to create the test level cost estimates in policy; and (d) break out funding requests by test in the BMDS Accountability Report and other budget documentation submitted during the annual budget submission.

Agency: Department of Defense
Status: Open

Comments: The department partially concurred with our recommendation, agreeing to include a detailed crosswalk of changes to each test in the Ballistic Missile Defense System (BMDS) Integrated Master Test Plan. However, DOD did not concur with the remaining three parts of our recommendation that include steps related to improving scheduling, cost, and reporting on MDA's test program. In August 2020, we observed that MDA had taken actions on our recommendation, such as including more detailed information on changes to the test schedule in its 2018 and 2019 versions of the BMDS Integrated Master Test Plan. We have an ongoing review to assess MDA's program and test cost estimates and plan to review the BMDS Integrated Master Test Plan to determine if the full intent of our recommendation is being met.

Recommendation: To strengthen MDA's acquisition efforts and strengthen oversight, and to improve MDA's requirement-setting process and ensure it includes an appropriate balance between MDA and warfighter priorities, the Secretary of Defense should require MDA to develop a plan to transition operational requirements analysis currently performed within MDA's Achievable Capabilities List to the U.S. Combatant Commanders, with U.S. Strategic Command as the lead entity and, in the interim, require MDA to obtain their concurrence of the Achievable Capabilities List prior to its release.

Agency: Department of Defense
Status: Open

Comments: DOD did not concur with our recommendation to allow the warfighter to determine operational-level requirements for the Ballistic Missile Defense System (BMDS). Although the department disagreed with our recommendation, the Director, MDA informed us in March 2018 that he supported the warfighter providing MDA with operational-level BMDS requirements, provided they are approved by the Combatant Commander for U.S. Strategic Command. The Director, MDA also agreed to obtain U.S. Strategic Command's concurrence on the Achievable Capabilities List prior to its release. Moreover, the January 2019 Missile Defense Review clarified that missile defense requirements are established through the Warfighter Involvement Process, which is governed by U.S. Strategic Command. The review also directed DOD components to evaluate the current missile defense requirements process to ensure that Service and Combatant Commanders' involvement occurs as early as possible in the capabilities development process. According to a U.S. Strategic Command official, in July 2019, the Missile Defense Executive Board agreed with a working group's finding that improvements to the process were needed. In August 2020, U.S. Strategic Command released an update to its instruction that articulates the BMDS Warfighter Involvement Process to address issues identified by the working group. The Director, MDA has also stated in a March 2020 congressional hearing that requirements for the Next Generation Interceptor were coordinated with combatant commanders and the Joint Requirements Oversight Council, which is a significant and positive development. We intend to evaluate these efforts, as they may potentially satisfy the actions we included in our recommendation.

Recommendation: To mitigate the risk of poor acquisition outcomes and strengthen the department's investment decisions, the Secretary of Homeland Security should direct the Undersecretary for Management to update the acquisition policy to require that major acquisition programs' technical requirements are well defined and key technical reviews are conducted prior to approving programs to initiate product development and establishing Acquisition Program Baselines, in accordance with acquisition best practices.

Agency: Department of Homeland Security
Status: Open

Comments: In providing comments on this report, DHS concurred with our recommendation and stated that it planned to initiate a study to assess how to better align its processes for technical reviews and acquisition decisions. Upon completion of the study, DHS updated its acquisition policy instruction in May 2019, which adjusted the acquisition life cycle. Specifically, the updated instruction requires programs to conduct key technical reviews--including a preliminary design review--prior to establishing the program's Acquisition Program Baseline. As of July 2020, DHS was in the process of updating the related policies and guidance for its Systems Engineering Life Cycle, which govern the department's technical reviews. GAO will review these policies and guidance once complete to confirm alignment with the changes made to the acquisition management instruction and will monitor DHS's implementation of its new acquisition life cycle to ensure the department's actions meet the intent of this recommendation.

Recommendation: The Secretary of Commerce and Under Secretary for Economic Affairs should direct the Census Bureau to revise and test procedures on how to better leverage enumerator-collected information on the best time or day to conduct interviews, and ensure enumerators are properly trained on these procedures.

Agency: Department of Commerce
Status: Open

Comments: Commerce agreed with this recommendation. In the Bureau's March 2017 action plan, it reported that as the Census Bureau develops enhancements to the procedures and training for enumerators, it will reflect on what it has learned from 2016 Census Test experiences as well as from recommendations such as this. It reported a target completion of December 2018, following completion and analyses of the nonresponse follow-up operation for the 2018 End-to-End Census Test. In January 2018, Bureau officials told us that leveraging enumerator collected information on the best time to conduct an interview would not be a part of questionnaire design's functionality for 2020 and will most likely rely on the system optimizer to determine the best time to contact a household. Bureau officials indicated they may push this recommendation out to 2030. During the 2018 Test, we continued to observe that enumerator notes were not being systematically reviewed by supervisors and managers, such that information such as office hours for apartment managers among other information we reported on was not being used by the operation. In June 2019, the Bureau informed us that it had added functionality to the enumerator device for enumerators to alert supervisors of case notes of "high importance" and was revising training to explain its use. Additionally, in December 2019, the Bureau shared documentation that reiterated the importance of enumerators reviewing prior case notes but clarified that enumerators should not expect their supervisors to regularly review those case notes. In April 2020, Bureau officials indicated that it was not likely that the Bureau would revise CFS training to systematically review case notes for the 2020 cycle, though we are following up with the Bureau to see if this can be done given the revised census time frames stemming from the COVID-19 outbreak. To fully implement this recommendation for 2020 and for future fieldwork, the Bureau needs to make better use of the information collected by enumerators during interview attempts about when to make additional attempts, such as during reported working hours of property managers for large multi-unit structures that house a large number of non-respondents.

Recommendation: In the event that operational test results for PDB-8 and PDB-8.1 reveal performance shortfalls that require additional development of the near and mid-term upgrades tested, the Secretary of Defense should direct the Secretary of the Army to establish mechanisms for overseeing those upgrades commensurate with other major defense acquisition programs, to include an initial report--similar to a Selected Acquisition Report--as soon as practical following operational testing for both PDB-8 and PDB-8.1, on the near and mid-term upgrades evaluated during these tests, including: (1) cost, schedule, and performance estimates for any additional development that is needed; and (2) an estimate of the amount of development costs it has incurred since 2013 for near- and mid-term Patriot upgrades operationally tested along with PDB-8 and PDB-8.1.

Agency: Department of Defense
Status: Open

Comments: DOD partially concurred with this recommendation. In 2017, operational testing for PDB-8 was completed and results show some performance shortfalls although DOD asserts that there is no additional development required. DOD plans to reassess the need for any additional action after operational testing for PDB-8.1 is completed as planned in the first quarter of fiscal year 2022.

Recommendation: In the event that operational test results for PDB-8 and PDB-8.1 reveal performance shortfalls that require additional development of the near and mid-term upgrades tested, the Secretary of Defense should direct the Secretary of the Army to establish mechanisms for overseeing those upgrades commensurate with other major defense acquisition programs, to include annual updates to Congress comparing the latest cost and schedule estimates against the initial estimates and providing explanations for any major deviations until development is complete.

Agency: Department of Defense
Status: Open

Comments: DOD partially concurred with this recommendation. In 2017, operational testing for PDB-8 was completed and results show some performance shortfalls although DOD asserts that there is no additional development required. DOD plans to reassess the need for any additional action after operational testing for PDB-8.1 is completed as planned in the first quarter of fiscal year 2022.

Recommendation: To more accurately communicate DHS's funding plans for USCG's major acquisition programs, the Secretary of Homeland Security should ensure the funding plans presented to Congress in fiscal year 2015 are comprehensive and clearly account for all operations and maintenance funding DHS plans to allocate to each of the USCG's major acquisition programs.

Agency: Department of Homeland Security
Status: Open

Comments: The Department of Homeland Security (DHS) concurred with this recommendation and stated in their comments that the U.S. Coast Guard and the DHS Chief Financial Officer will develop a plan to address this recommendation by September 30, 2015, then work together to fully implement the plan. DHS estimated it would complete this effort March 31, 2016. However, the USCG encountered technical challenges during this process and was unable to implement the plan by that time. The U.S. Coast Guard has revised the estimated completion date and now anticipates it will be able to address this recommendation in fiscal year 2022. GAO will continue to assess the updated APBs as a part of its annual review of select DHS major acquisition programs to determine whether the department has addressed the recommendation.

Recommendation: To help DOD address key risks to F-35 affordability and operational readiness, and to improve the reliability of its O&S cost estimates for the life cycle of the program, the Secretary of Defense should direct the F-35 Program Executive Officer, to enable DOD to better identify, address, and mitigate performance issues with the Autonomic Logistics Information System (ALIS) that could have an effect on affordability, as well as readiness, to establish a performance-measurement process for ALIS that includes, but is not limited to, performance metrics and targets that (1) are based on intended behavior of the system in actual operations and (2) tie system performance to user requirements.

Agency: Department of Defense
Status: Open
Priority recommendation

Comments: DOD concurred with our recommendation. According to DOD officials, as of July 2018, plans are emerging between the services and the Joint Program Office on a path forward for ALIS, focusing on both the current iteration of ALIS and the future state. Going forward, the services and the Joint Program Office are developing plans for the necessary re-architecture of ALIS. Once these current improvements and future requirements are finalized, appropriate performance metrics, tying system performance in operations environments to user requirements, will be incorporated. As of January 2020, DOD officials stated that there was no update to this status. Although DOD has a way ahead as it relates to developing performance metrics for ALIS, DOD has yet to develop any metrics that are based on intended behavior of the system and tie system performance to user requirements. Until DOD takes this action, our recommendation will remain open.

Recommendation: To help DOD address key risks to F-35 affordability and operational readiness, and to improve the reliability of its O&S cost estimates for the life cycle of the program, the Secretary of Defense should direct the F-35 Program Executive Officer, to develop a high level of confidence that the aircraft will achieve its R+M goals, to develop a software reliability and maintainability (R+M) assessment process, with metrics, by which the program can monitor and determine the effect that software issues may have on overall F-35 R+M issues.

Agency: Department of Defense
Status: Open

Comments: DOD concurred with our recommendation. According to DOD officials, as of July 2018, the department and the Joint Program Office, as part of their focus on agile software development, are working to incorporate software reliability and maintainability metrics into future software development and sustainment contracts. Some of the proposed metrics under consideration include: change failure rate; number of errors in developmental/user/operational testing; time to fix on critical errors; and mean time to restore. As of September 2019, DOD officials stated that there was no update to this status. Although attention is being paid to software Reliability & Maintainability, until DOD develops a process focused on software and its effects on overall Reliability & Maintainability issues, this recommendation will remain open.

Recommendation: To help DOD address key risks to F-35 affordability and operational readiness, and to improve the reliability of its O&S cost estimates for the life cycle of the program, the Secretary of Defense should direct the F-35 Program Executive Officer, to promote competition, address affordability, and inform its overarching sustainment strategy, to develop a long-term Intellectual Property (IP) Strategy to include, but not be limited to, the identification of (1) current levels of technical data rights ownership by the federal government and (2) all critical technical data needs and their associated costs.

Agency: Department of Defense
Status: Open
Priority recommendation

Comments: DOD concurred with our recommendation. According to DOD officials, as of January 2020, in the updated F-35 Life Cycle Sustainment Plan (LCSP) issued in January 2019, "Secure Use of Appropriate Technical Data" was one of the identified elements of success necessary to improve F-35 readiness and reduce sustainment costs. As part of the ongoing Plans of Action & Milestones (POAM) implementation process for the LCSP success elements, the F-35 Joint Program Office is working with the OEMs to determine the data rights the government already has, and to determine the specific technical data the government needs, and what it needs that data to accomplish. Significant progress has been made on both fronts with the prime contractor. We acknowledge that progress surrounding technical data rights is being made; however, until an Intellectual Property strategy is developed and released, this recommendation will remain open.

Recommendation: To provide the Congress with the necessary insight into program planning and affordability, and to decrease the risk of cost and schedule overruns, NASA's Administrator should direct the Human Exploration and Operations Mission Directorate to take the following action: To allow for a continued assessment of progress and affordability, NASA should structure each future increment of SLS capability with a total cost exceeding the $250 million threshold for designation as a major project as a separate development effort within the SLS program. In doing so, NASA should require each increment to complete both the technical and programmatic reviews required of other major development projects, per the agency's acquisition and system engineering policies.

Agency: National Aeronautics and Space Administration
Status: Open
Priority recommendation

Comments: NASA agreed with this recommendation. In January 2020, NASA stated that it planned to establish an agency baseline commitment for capability upgrades (e.g., Block 1B upgrades, such as Mobile Launcher-2 and Exploration Upper Stage) above the $250 million threshold. A joint confidence level analysis will be performed at key decision points and will include the cost and schedule range estimates for each of these upgrades. To fully implement this recommendation, however, NASA needs to provide evidence that each capability upgrade is designated a major project and is required to complete the technical and programmatic reviews required of other major development projects.

Recommendation: To provide the Congress with the necessary insight into program planning and affordability, and to decrease the risk of cost and schedule overruns, NASA's Administrator should direct the Human Exploration and Operations Mission Directorate to take the following action: Provide decision makers with an informed basis for making investment decisions regarding the SLS program, NASA should identify a range of possible missions for each future SLS variant that includes cost and schedule estimates and plans for how those possible missions would fit within NASA's funding profile.

Agency: National Aeronautics and Space Administration
Status: Open
Priority recommendation

Comments: NASA agreed with this recommendation. In January 2020, NASA stated that new leadership at the Human Exploration and Operations Mission Directorate is conducting an internal independent assessment to assess the current schedule and technical approach for achieving a lunar landing by 2024, including the utilization of SLS for Artemis missions. With the insights gained from this assessment, NASA will move forward with planning and executing these missions through the annual budgeting process. To fully address this recommendation, NASA will need to identify cost and schedule estimates for possible SLS missions beyond its first exploration mission, now known as Artemis I, and how its planned missions would fit within NASA's funding profile.

Recommendation: To provide the Congress with the necessary insight into program affordability, ensure its ability to effectively monitor total program costs and execution, and to facilitate investment decisions, the NASA's Administrator should direct the Human Exploration and Operations Mission Directorate to establish a separate cost and schedule baseline for work required to support the SLS Block I Exploration Mission-2 (EM-2) and report this information to the Congress through NASA's annual budget submission. If NASA decides to fly the SLS Block I beyond EM-2, establish separate life cycle cost and schedule baseline estimates for those efforts, to include funding for operations and sustainment, and report this information annually to Congress via the agency's budget submission.

Agency: National Aeronautics and Space Administration
Status: Open
Priority recommendation

Comments: NASA partially agreed with this recommendation, stating that it defined and documented life-cycle costs for SLS to a first demonstrated capability, consistent with cost estimating best practices and NASA project and program management policy and that it would report costs associated with the second exploration mission via its annual budget submission. In January 2020, NASA stated that it is evaluating changes to NASA Procedural Requirements 7120.5E, "NASA Space Flight Program and Project Management Requirements," to better enable the necessary insight into program affordability and efficient monitoring of total program costs and execution for multi-year, multi-cadence type programs. Further, NASA stated that it is investigating plans to redefine performance expectations for multi-decade programs' formal commitments while maintaining visibility to the entire plan. To address this recommendation, NASA needs to establish separate cost and schedule baselines for work required to support SLS for EM-2, which is now known as Artemis II.

Recommendation: To provide the Congress with the necessary insight into program affordability, ensure its ability to effectively monitor total program costs and execution, and to facilitate investment decisions, because NASA intends to use the increased capabilities of the SLS, Orion, and Ground Systems Development and Operations efforts well into the future and has chosen to estimate costs associated with achieving the capabilities, the NASA's Administrator should direct the Human Exploration and Operations Mission Directorate to establish separate cost and schedule baselines for each additional capability that encompass all life cycle costs, to include operations and sustainment. When NASA cannot fully specify costs due to lack of well-defined missions or flight manifests, forecast a cost estimate range -- including life cycle costs -- having minimum and maximum boundaries. These baselines or ranges should be reported to Congress annually via the agency's budget submission.

Agency: National Aeronautics and Space Administration
Status: Open
Priority recommendation

Comments: NASA partially agreed with this recommendation, stating that it had established separate programs for SLS, Orion, and the ground systems and adopted a block upgrade approach for SLS. In January 2020, NASA stated that it plans to establish an agency baseline commitment for capability upgrades (e.g., Block 1B upgrades, such as Mobile Launcher-2 and Exploration Upper Stage) above the $250 million threshold. A joint confidence level analysis will be performed at key decision points and will include the cost and schedule range estimates for each of these upgrades. To address this recommendation, NASA needs to provide evidence that it established separate cost and schedule baselines for each additional SLS, Orion, and Ground Systems Development and Operations capability blocks that encompass all life-cycle costs, including operations and sustainment.

Recommendation: To allow Congress to better oversee management of the nuclear security enterprise and to improve NNSA's management information with respect to the base capabilities necessary to ensure nuclear weapons are safe, secure, and reliable, the Administrator of NNSA should, once the Stockpile Services work breakdown structure reflects a product or capability basis, use this work breakdown structure to develop product/capability cost estimates that adequately justify the congressional budget request for Stockpile Services.

Agency: Department of Energy: National Nuclear Security Administration
Status: Open

Comments: NNSA has significantly improved its work breakdown structure for Stockpile Services. The work breakdown structure now reflects a product or capability basis to a much greater extent than it did previously. NNSA restructured its budget starting with its fiscal year 2021 budget justification materials. GAO, through ongoing work, is continuing to monitor NNSA's development of cost estimates for Stockpile Services products and capabilities that inform future years' budget requests and justifications.

Recommendation: To improve the Bureau's use of its master schedule to manage the 2020 decennial census, the Secretary of Commerce should require the Director of the U.S. Census Bureau to include estimates of the resources, such as labor, materials, and overhead costs, in the 2020 integrated schedule for each activity as the schedule is built, and prepare to carry out other steps as necessary to conduct systematic schedule risk analyses on the 2020 schedule.

Agency: Department of Commerce
Status: Open
Priority recommendation

Comments: Commerce neither agreed nor disagreed with this recommendation. Regarding GAO's 2013 assessment of the Bureau's schedule (GAO-14-59), Bureau officials stated that they hoped to begin identifying the resources needed for each activity in their schedules by early 2014. Bureau officials announced they had completed the 2020 Census schedule in July 2016, and have since periodically described their intent to link resources to activities within their schedules. However, as of May 2018, when the Bureau had not taken these steps. Senior Bureau officials stated that it would require additional staffing in order to plan for and implement this recommendation. In July 2018 (GAO-18-589) we reported again on the status of the Bureau's scheduling, stating that when the Bureau has resource loaded its schedule, it will be able to use the schedule more effectively as a management tool. The Bureau took steps toward assigning resources to its master activity schedule for the 2020 Census, but effectively ran out of time to do so. Assigning resources to large complex schedules is easier to do early in schedule development process, as we recommended the Bureau do in 2009 for its 2020 Census schedule. This recommendation will remain open pending the Bureau taking steps in developing its 2030 schedule with appropriate resources linked to it.