Reports & Testimonies

  • GAO’s recommendations database contains report recommendations that still need to be addressed.

    GAO’s recommendations help congressional and agency leaders prepare for appropriations and oversight activities, as well as help improve government operations. Recommendations remain open until they are designated as Closed-implemented or Closed-not implemented. You can explore open recommendations by searching or browsing.

    GAO's priority recommendations are those that we believe warrant priority attention. We sent letters to the heads of key departments and agencies, urging them to continue focusing on these issues. These recommendations are labeled as such. You can find priority recommendations by searching or browsing our open recommendations below, or through our mobile app.

  • Browse Open Recommendations

    Explore priority recommendations by subject terms or browse by federal agency

    Search Open Recommendations

    Search for a specific priority recommendation by word or phrase



  • Governing on the go?

    Our Priorities for Policy Makers app makes it easier for leaders to search our recommendations on the go.

    See the November 10th Press Release


  • Have a Question about a Recommendation?

    • For questions about a specific recommendation, contact the person or office listed with the recommendation.
    • For general information about recommendations, contact GAO's Audit Policy and Quality Assurance office at (202) 512-6100 or apqa@gao.gov.
  • « Back to Results List Sort by   

    Results:

    Subject Term: "Operational testing"

    36 publications with a total of 94 open recommendations including 13 priority recommendations
    Director: Robert Goldenkoff
    Phone: (202) 512-2757

    3 open recommendations
    Recommendation: The Secretary of Commerce should direct the Under Secretary of the Economics and Statistics Administration and the Director of the U.S. Census Bureau to use the Bureau's evaluations before 2020 to determine the implications of in-office address canvassing on the cost and quality of address canvassing, and use this information to justify decisions related to its re-engineered address canvassing approach.

    Agency: Department of Commerce
    Status: Open

    Comments: When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.
    Recommendation: Early in the next decennial cycle, the Secretary of Commerce should direct the Under Secretary of the Economics and Statistics Administration and the Director of the U.S. Census Bureau to plan and execute more flexible, and perhaps smaller, address canvassing test and evaluation activity needed to support key design decisions having significant effect on the cost and quality of the census.

    Agency: Department of Commerce
    Status: Open

    Comments: When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.
    Recommendation: The Secretary of Commerce should direct the Under Secretary of the Economics and Statistics Administration and the Director of the U.S. Census Bureau to use productivity measures that track the progress in completing in-office address canvassing workload and the effectiveness of in-office address canvassing in reducing fieldwork in order to make informed decisions on allocating resources to current and future address canvassing workload so that the operation is completed in a timely and cost-effective manner.

    Agency: Department of Commerce
    Status: Open

    Comments: When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.
    Director: Michele Mackin
    Phone: (202) 512-4841

    4 open recommendations
    Recommendation: The Secretary of Defense should direct the Secretary of the Navy to revise the Navy's ship delivery policy to clarify what types of deficiencies need to be corrected and what mission capability (including the levels of quality and capability) must be achieved at (1) delivery and (2) when the ship is provided to the fleet (at the obligation work limiting date (OWLD)). In doing so, the Navy should clearly define what constitutes a complete ship and when that should be achieved.

    Agency: Department of Defense
    Status: Open

    Comments: When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.
    Recommendation: The Secretary of Defense should direct the Secretary of the Navy to reconcile policy with practice to support INSURV's role in making a recommendation for fleet introduction. Accomplishing this may require a study of the current timing of ship trials, and the costs and benefits associated with adding an INSURV assessment prior to providing ships to the fleet.

    Agency: Department of Defense
    Status: Open

    Comments: When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.
    Recommendation: The Secretary of Defense should direct the Secretary of the Navy to reflect additional ship milestones in Selected Acquisition Reports to Congress, including OWLD and readiness to deploy.

    Agency: Department of Defense
    Status: Open

    Comments: When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.
    Recommendation: The Secretary of Defense should direct the Secretary of the Navy to, in Selected Acquisition Reports to Congress, ensure that the criteria used to declare IOC aligns with DOD guidance, and reflect the definition of this milestone in the reports.

    Agency: Department of Defense
    Status: Open

    Comments: When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.
    Director: Cristina Chaplain
    Phone: (202) 512-4841

    4 open recommendations
    Recommendation: To strengthen MDA's acquisition efforts and strengthen oversight, and to increase traceability and insight into MDA's test program, the Secretary of Defense should require MDA to (a) include a detailed crosswalk of changes to each test, such as names, planned execution dates, test types, targets, and other modifications, in each iteration of its Integrated Master Test Plan; (b) address deficiencies in its test scheduling policy by better aligning it with best practices for scheduling, including the use of a schedule work breakdown structure (WBS) that clearly traces each activity to the cost WBS, properly assigning resources to schedules, and clarifying guidance on when and how to conduct schedule risk analysis; (c) rectify deficiencies in its element and test level cost estimates by requiring the use of the common test WBS, documenting the traceability of source data, and codifying the processes and associated information for the software application Test Resource Mission Planning-Tool used to create the test level cost estimates in policy; and (d) break out funding requests by test in the BMDS Accountability Report and other budget documentation submitted during the annual budget submission.

    Agency: Department of Defense
    Status: Open

    Comments: When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.
    Recommendation: To strengthen MDA's acquisition efforts and strengthen oversight, and to improve MDA's requirement-setting process and ensure it includes an appropriate balance between MDA and warfighter priorities, the Secretary of Defense should require MDA to develop a plan to transition operational requirements analysis currently performed within MDA's Achievable Capabilities List to the U.S. Combatant Commanders, with U.S. Strategic Command as the lead entity and, in the interim, require MDA to obtain their concurrence of the Achievable Capabilities List prior to its release.

    Agency: Department of Defense
    Status: Open

    Comments: When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.
    Recommendation: To strengthen MDA's acquisition efforts and strengthen oversight, and to ensure that the Redesigned Kill Vehicle (RKV) acquisition strategy continues to remain viable, promotes effective competition, and addresses concerns raised by DOD components, the Secretary of Defense should require the Director, DOD's Office of Cost Assessment and Program Evaluation (CAPE) to perform a comprehensive review of the RKV acquisition strategy and provide any recommendations to the Secretary of Defense that the Director deems necessary and appropriate to obtain CAPE's concurrence for the RKV program's acquisition strategy. Any decision to award a full-rate production contract should be delayed until after MDA has received approval from the Under Secretary of Defense for Acquisition, Technology, and Logistics (USD(AT&L) to proceed to full-rate production.

    Agency: Department of Defense
    Status: Open

    Comments: When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.
    Recommendation: To strengthen MDA's acquisition efforts and strengthen oversight, and to ensure that future acquisition strategies MDA develops for its new efforts reflect an appropriate balance between timeliness, affordability, reliability, and effectiveness and achieve department-wide buy-in, the Secretary of Defense should require MDA to produce acquisition strategies for all its major new efforts that are subject to review by the Director, CAPE and review and approval by the USD(AT&L).

    Agency: Department of Defense
    Status: Open

    Comments: When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.
    Director: Michael J. Sullivan
    Phone: (202) 512-4841

    3 open recommendations
    Recommendation: To ensure that DOD adequately prioritizes its resources to finish F-35 baseline development and delivers all of the promised warfighting capabilities and that Congress is fully informed when making fiscal year 2018 budget decisions, the Secretary of Defense should reassess the additional cost and time needed to complete developmental testing using historical program data.

    Agency: Department of Defense
    Status: Open

    Comments: The agency has not taken any action to implement this recommendation.
    Recommendation: To ensure that DOD adequately prioritizes its resources to finish F-35 baseline development and delivers all of the promised warfighting capabilities and that Congress is fully informed when making fiscal year 2018 budget decisions, the Secretary of Defense should delay the issuance of the Block 4 development request for proposals at least until developmental testing is complete and all associated capabilities have been verified to work as intended.

    Agency: Department of Defense
    Status: Open

    Comments: The agency has not taken any action to implement this recommendation.
    Recommendation: To ensure that DOD adequately prioritizes its resources to finish F-35 baseline development and delivers all of the promised warfighting capabilities and that Congress is fully informed when making fiscal year 2018 budget decisions, the Secretary of Defense should finalize the details of DOD and contractor investments associated with an economic order quantities (EOQ) purchase in fiscal year 2018, and submit a report to Congress with the fiscal year 2018 budget request that clearly identifies the details, including costs and benefits of the finalized EOQ approach.

    Agency: Department of Defense
    Status: Open

    Comments: In providing comments on this report, the agency partially concurred with this recommendation but has not yet taken any actions necessary to implement it.
    Director: Michele Mackin
    Phone: (202) 512-4841

    3 open recommendations
    Recommendation: To mitigate the risk of poor acquisition outcomes and strengthen the department's investment decisions, the Secretary of Homeland Security should direct the Undersecretary for Management to update the acquisition policy to require that major acquisition programs' technical requirements are well defined and key technical reviews are conducted prior to approving programs to initiate product development and establishing Acquisition Program Baselines, in accordance with acquisition best practices.

    Agency: Department of Homeland Security
    Status: Open

    Comments: In providing comments on this report, DHS concurred with our recommendation and stated that it planned to initiate a study to assess how to better align its processes for technical reviews and acquisition decisions. Upon completion of the study, DHS plans to update its acquisition policies, as appropriate.
    Recommendation: To mitigate the risk of poor acquisition outcomes and strengthen the department's investment decisions, the Secretary of Homeland Security should direct the Undersecretary for Management to update the acquisition policy to specify that acquisition decision memorandums clearly document the rationale of decisions made by DHS leadership, such as, but not limited to, the reasons for allowing programs to deviate from the requirement to obtain department approval for certain documents at Acquisition Decision Events and the results of considerations or trade-offs.

    Agency: Department of Homeland Security
    Status: Open

    Comments: In providing comments on this report, DHS concurred with our recommendation and stated that it had begun expanding the content included in Acquisition Decision Memorandums (ADM) to include greater detail and that future ADMs would address the status of the acquisition documentation. DHS also said it had updated the guidance for writing ADMs in a handbook for Office of Program Accountability and Risk Management staff, thus making progress toward satisfying the recommendation. However, we did not close the recommendation because the updated guidance was not incorporated into the department's official acquisition policy, which may limit DHS's ability to implement the changes consistently over time. We will continue to review ADMs to assess whether the department's actions address the intent of this recommendation.
    Recommendation: To mitigate the risk of poor acquisition outcomes and strengthen the department's investment decisions, the Secretary of Homeland Security should direct the Undersecretary for Management to update the acquisition policy to specify at what point minimum standards for KPPs should be met, and clarify the performance data that should be used to assess whether or not a performance breach has occurred.

    Agency: Department of Homeland Security
    Status: Open

    Comments: In providing comments on this report, DHS concurred with our recommendation and stated that it had updated guidance related to its performance breaches in a handbook for Office of Program Accountability and Risk Management staff, thus moving toward satisfying the intent of this recommendation. Specifically, DHS identified that programs' KPPs should be met and verified no later than initial operational test and evaluation conducted prior to Acquisition Decision Event 3, the point at which DHS leadership approves the program to transition into sustainment. If programs have not met a KPP by this point, they will be required to declare a performance breach and submit a remediation plan documenting the root cause of the breach, along with how and when the breach will be resolved. However, we did not close the recommendation because the department's official acquisition policy has yet to be updated. DHS will fully address this recommendation when it incorporates the changes into its acquisition policy to ensure that the updated guidance on performance breaches is communicated and implemented consistently throughout the department.
    Director: Robert Goldenkoff
    Phone: (202) 512-2757

    4 open recommendations
    including 1 priority recommendation
    Recommendation: The Secretary of Commerce and Under Secretary for Economic Affairs should direct the Census Bureau to determine the cause(s) for non-interviews experienced during the non-response follow-up operation and revise and test what, if any, changes need to be made to operational procedures, training, or both, including making contact with proxy respondents.

    Agency: Department of Commerce
    Status: Open
    Priority recommendation

    Comments: Commerce agreed with this recommendation. In the Bureau's March 2017 action plan, it reported that analysis of the 2016 Census Test data is underway and that the 2016 Census Test experiences and the ongoing 2016 Census Test data analysis will inform the Bureau's evolution of operational methods and development of revised training content and provided a target completion date of December 2018, following completion and analyses of the nonresponse follow-up operation for the 2018 End-to-End Census Test. Evaluation data and studies of the 2016 Census Test experience could provide insight on what operational procedures, training, business rules, or other circumstances resulted in the higher-than-expected number of non-interviews. Making changes as needed, if any, in advance of the 2018 End-to-End Test would provide an opportunity to test them and better ensure that high rates of non-interviews do not adversely affect 2020 Census data quality.
    Recommendation: The Secretary of Commerce and Under Secretary for Economic Affairs should direct the Census Bureau to revise and test operational procedures for accessing incomplete closed cases and revise and test training material to reflect when this flexibility to access incomplete closed cases should be used by the enumerator.

    Agency: Department of Commerce
    Status: Open

    Comments: Commerce agreed with this recommendation. In the Bureau's March 2017 action plan, it reported that the Bureau had identified enhancements to methods to address this recommendation and would consider how procedures can provide flexibility, and how the enumerator training can evolve to communicate the availability of this flexibility, and when the capability should be leveraged. It reported that how the software solutions will reflect these enhancements will inform the development of the procedures and training used by enumerators for the 2018 Census End-to-End Census Test. It reported a target completion of December 2018 following completion and analyses of the nonresponse follow-up operation for the 2018 End-to-End Census Test.
    Recommendation: The Secretary of Commerce and Under Secretary for Economic Affairs should direct the Census Bureau to revise and test operational procedures and relevant training materials for initial property manager visits to ensure procedures and training material are communicated to and understood by enumerators and their supervisors.

    Agency: Department of Commerce
    Status: Open

    Comments: Commerce agreed with this recommendation. In the Bureau's March 2017 action plan, it reported that based on its 2016 Census Test experiences, it had identified modifications to the system capabilities and procedures for initial multiunit manager visits and that it would implement and further study these changes in future tests, including the 2018 End-to-End Census Test. It reported a target completion in December 2018, following completion and analyses of the nonresponse follow-up operation for the 2018 End-to-End Census Test following completion and analyses of the nonresponse follow-up (NRFU) operation for the 2018 End-to-End Census Test.
    Recommendation: The Secretary of Commerce and Under Secretary for Economic Affairs should direct the Census Bureau to revise and test procedures on how to better leverage enumerator-collected information on the best time or day to conduct interviews, and ensure enumerators are properly trained on these procedures.

    Agency: Department of Commerce
    Status: Open

    Comments: Commerce agreed with this recommendation. In the Bureau's March 2017 action plan, it reported that as the Census Bureau develops enhancements to the procedures and training for enumerators, it will reflect on what it has learned from 2016 Census Test experiences as well as from recommendations such as this. It reported a target completion of December 2018, following completion and analyses of the nonresponse follow-up operation for the 2018 End-to-End Census Test.
    Director: David Trimble
    Phone: (202) 512-3841

    1 open recommendations
    Recommendation: To help ensure that NNSA can better inform long-term planning and management decision making as well as to ensure that the Enhanced Surveillance Program complements NNSA's other efforts to assess the nuclear weapons stockpile, the NNSA Administrator should develop a long-term strategy for the Enhanced Surveillance Program that incorporates outcome-oriented strategic goals, addresses management challenges and identifies resources needed to achieve these goals, and develops and uses performance measures to track progress in achieving these goals.

    Agency: Department of Energy: National Nuclear Security Administration
    Status: Open

    Comments: Fiscal year 2018 budget submission data indicates some new activities but not yet mention a strategy change.
    Director: Cristina Chaplain
    Phone: (202) 512-4841

    2 open recommendations
    Recommendation: In the event that operational test results for PDB-8 and PDB-8.1 reveal performance shortfalls that require additional development of the near and mid-term upgrades tested, the Secretary of Defense should direct the Secretary of the Army to establish mechanisms for overseeing those upgrades commensurate with other major defense acquisition programs, to include an initial report--similar to a Selected Acquisition Report--as soon as practical following operational testing for both PDB-8 and PDB-8.1, on the near and mid-term upgrades evaluated during these tests, including: (1) cost, schedule, and performance estimates for any additional development that is needed; and (2) an estimate of the amount of development costs it has incurred since 2013 for near- and mid-term Patriot upgrades operationally tested along with PDB-8 and PDB-8.1.

    Agency: Department of Defense
    Status: Open

    Comments: DOD partially concurred with this recommendation, however, it is too early to determine what, if any, actions the agency will take until the results of operational testing for PDB-8 are made available following its planned completion in late summer 2017.
    Recommendation: In the event that operational test results for PDB-8 and PDB-8.1 reveal performance shortfalls that require additional development of the near and mid-term upgrades tested, the Secretary of Defense should direct the Secretary of the Army to establish mechanisms for overseeing those upgrades commensurate with other major defense acquisition programs, to include annual updates to Congress comparing the latest cost and schedule estimates against the initial estimates and providing explanations for any major deviations until development is complete.

    Agency: Department of Defense
    Status: Open

    Comments: DOD partially concurred with this recommendation, however, it is too early to determine what, if any, actions the agency will take until the results of operational testing for PDB-8 are made available following its planned completion in late summer 2017.
    Director: Michele Mackin
    Phone: (202) 512-4841

    2 open recommendations
    Recommendation: To ensure a more accurate estimate of the expected cost savings under the fiscal year 2013-2017 multiyear procurement, Congress should consider requiring the Navy to update its estimate of savings, which currently reflects only Flight IIA ships, to increase transparency for costs and savings for Congress and the taxpayers, as well as provide improved information to support future multiyear procurement savings estimates.

    Agency: Congress
    Status: Open

    Comments: To ensure a more accurate estimate of the expected cost savings under the fiscal year 2013-2017 multiyear procurement, we asked Congress to consider requiring the Navy to update its estimate of savings, which currently reflects only Flight IIA ships, to increase transparency for costs and savings for Congress and the taxpayers, as well as provide improved information to support future multi-year procurement savings estimates. Neither the Senate nor House National Defense Authorization Act (NDAA) reports for fiscal year 2018 direct the Navy to update its savings and both reports include language authorizing the Navy to pursue a DDG 51 Flight III multi-year procurement contract for fiscal years 2018-2022. We will continue to monitor the status of this matter at least until the NDAA for fiscal year 2018 is enacted, at which time we will close the matter as not implemented if the multi-year procurement is authorized and no savings update requirement is included.
    Recommendation: To better support DDG 51 Flight III oversight, the Secretary of Defense should designate the Flight III configuration as a major subprogram of the DDG 51 program in order to increase the transparency, via Selected Acquisition Reports, of Flight III cost, schedule, and performance baselines within the broader context of the DDG 51 program.

    Agency: Department of Defense
    Status: Open

    Comments: DOD agreed that visibility into DDG 51 Flight III cost, schedule, and performance is important for oversight, but does not plan to designate Flight III as a major subprogram. No further DOD action has been taken on this recommendation and congressional reports supporting the National Defense Authorization Act for Fiscal Year 2018--yet to be finalized and enacted--do not include any direction for the department to do so. Nevertheless, with construction of the lead Flight III ship only recently awarded (June 2017), we will continue to monitor any action taken to designate Flight III as a major subprogram.
    Director: Carol C. Harris
    Phone: (202) 512-4456

    9 open recommendations
    Recommendation: To provide reasonable assurance that the program executes Agile software development for USCIS ELIS consistent with its own policies and guidance and follows applicable leading practices, the Secretary of the Department of Homeland Security (DHS) should direct the Director of USCIS to direct the USCIS Chief Information Officer (CIO), in coordination with the DHS CIO and the Chief of the Office of Transformation Coordination (OTC), to review and update, as needed, existing policies and guidance and consider additional controls to complete planning for software releases prior to initiating development and ensure software meets business expectations prior to deployment.

    Agency: Department of Homeland Security
    Status: Open

    Comments: As of July 2017, the U.S. Citizenship and Immigration Services (USCIS) within the Department of Homeland Security (DHS) had taken steps to address this recommendation. In particular, in June 2017, USCIS provided an updated policy, dated April 2017, governing planning and deploying software releases. USCIS also demonstrated partial compliance with that policy. For example, it provided some release planning review documentation for recent releases that are required by the updated policy, including readiness review memos for releases 7.2 and 8.1. However, USCIS did not demonstrate that the program responsible for developing the USCIS Electronic Immigration System (USCIS ELIS) was consistently following its updated policy. For example, USCIS did not demonstrate that the program was completing all planning activities prior to initiating development, as called for in its updated policy. Moreover, the agency did not demonstrate compliance with its previous policy for all software releases planned and deployed since our July 2016 report. We will continue to work with USCIS to monitor actions the agency is taking to address this recommendation.
    Recommendation: To provide reasonable assurance that the program executes Agile software development for USCIS ELIS consistent with its own policies and guidance and follows applicable leading practices, the Secretary of DHS should direct the Director of USCIS to direct the USCIS CIO, in coordination with the DHS CIO and the Chief of OTC, to review and update, as needed, existing policies and guidance and consider additional controls to consistently implement the principles of the framework adopted for Agile software development.

    Agency: Department of Homeland Security
    Status: Open

    Comments: As of July 2017, USCIS had taken steps to address this recommendation. For example, in May 2017, USCIS provided updated policy governing the development of software releases, dated April 2017, along with release planning artifacts specific to USCIS ELIS. The updated policy included an appendix devoted to generally accepted agency practices and applying Agile principles in the agency. However, USCIS had not clearly indicated if USCIS ELIS was to implement the practices described in the policy. For example, the updated policy did not require program compliance with the generally accepted agency practices. Moreover, supporting artifacts from the release planning process did not always define a commitment to a particular development methodology or set of development practices. For example, the team process agreements, which describe how members of individual teams will work with each other, did not indicate if developers were to adhere to the practices described in updated USCIS policy. We will continue to work with USCIS to obtain additional documentation about actions it is taking to address this recommendation.
    Recommendation: To provide reasonable assurance that the program executes Agile software development for USCIS ELIS consistent with its own policies and guidance and follows applicable leading practices, the Secretary of DHS should direct the Director of USCIS to direct the USCIS CIO, in coordination with the DHS CIO and the Chief of OTC, to review and update, as needed, existing policies and guidance and consider additional controls to define and consistently execute appropriate roles and responsibilities for individuals responsible for development activities consistent with its selected development framework.

    Agency: Department of Homeland Security
    Status: Open

    Comments: As of July 2017, USCIS had taken steps to address this recommendation. For example, in June 2017, USCIS provided updated policy, dated April 2017, governing the development of software releases and release planning artifacts. The updated policy and release documentation defined some roles and responsibilities that were previously only described by USCIS in its informal November 2014 management model, such as the authority and responsibility of a product owner. However, program documentation and policy did not define all of the roles and responsibilities. For example, program documentation and policy did not define the roles and responsibilities of a facilitator, or Scrum Master, which is a position identified in leading practices for software development using Scrum, the development methodology previously identified by the program. In addition, USCIS did not demonstrate that it had defined and committed to an updated development methodology for software releases. Such a defined methodology will impact expectations for the roles and responsibilities in software development. Without such a defined methodology or approach to Agile software development, it is not clear if roles and responsibilities defined by previously documented approach to Agile software development are still applicable for the current development approach. Moreover, documentation associated with program releases and updated policy did not define all of the roles and responsibilities for positions described by USCIS in its May 2017 written response to GAO. We will continue to work with USCIS to obtain additional documentation about actions it is taking to address this recommendation.
    Recommendation: To provide reasonable assurance that the program executes Agile software development for USCIS ELIS consistent with its own policies and guidance and follows applicable leading practices, the Secretary of DHS should direct the Director of USCIS to direct the USCIS CIO, in coordination with the DHS CIO and the Chief of OTC, to review and update, as needed, existing policies and guidance and consider additional controls to identify all system users and involve them in release planning activities.

    Agency: Department of Homeland Security
    Status: Open

    Comments: As of July 2017, DHS and USCIS had not provided information demonstrating that the department has addressed this recommendation. In October 2016, DHS provided a written response stating that the USCIS Office of Information Technology and Office of Transformation Coordination were working closely with the various USCIS directorates to obtain and integrate feedback through regular review sessions with the end users and through additional end user testing. However, as of July 2017, DHS and USCIS have not provided new information about the status of this recommendation.
    Recommendation: To provide reasonable assurance that the program executes Agile software development for USCIS ELIS consistent with its own policies and guidance and follows applicable leading practices, the Secretary of DHS should direct the Director of USCIS to direct the USCIS CIO, in coordination with the DHS CIO and the Chief of OTC, to review and update, as needed, existing policies and guidance and consider additional controls to write user stories that identify user roles, include estimates of complexity, take no longer than one sprint to complete, and describe business value.

    Agency: Department of Homeland Security
    Status: Open

    Comments: As of July 2017, USCIS had provided GAO with documentation intended to demonstrate that the agency had taken steps to address this recommendation. For example, in May 2017, USCIS provided updated policy governing the development of software releases along with release planning artifacts specific to USCIS ELIS and an Independent Verification and Validation assessment. The agency also provided a series of backlogs that captured user stories for some software releases. In addition, the Independent Verification and Validation assessment indicated that the program was tracking user story quality as part of assessing whether value was continuously discovered and aligned to the mission. However, the assessment report provided to GAO indicated a negative trend for this outcome. Moreover, USCIS policy no longer set expectations regarding user story development. In addition, supporting artifacts from the release planning process did not always define a commitment to a particular development methodology, which is turn impacts the expectations for writing user stories. Finally, backlogs provided by USCIS did not cover all releases in development since our July 2016 report and did not include enough detail to assess all aspects of the user story process (e.g., story size and user involvement). We will continue to work with USCIS to obtain additional documentation about actions it is taking to address this recommendation.
    Recommendation: To provide reasonable assurance that the program executes Agile software development for USCIS ELIS consistent with its own policies and guidance and follows applicable leading practices, the Secretary of DHS should direct the Director of USCIS to direct the USCIS CIO, in coordination with the DHS CIO and the Chief of OTC, to review and update, as needed, existing policies and guidance and consider additional controls to establish outcomes for Agile software development.

    Agency: Department of Homeland Security
    Status: Open

    Comments: As of July 2017, USCIS had taken steps to address this recommendation. For example, in April 2017, USCIS issued updated policy governing software development at the agency. The updated policy included an appendix devoted to generally accepted agency practices and applying Agile principles in the agency. This appendix also included a set of ten outcomes associated with using Agile practices at USCIS. For example, outcomes included that value is continuously discovered and aligned to the mission. However, the updated policy did not require program compliance with the practices and principles described in the appendix. Moreover, the agency did not demonstrate that USCIS ELIS had committed to achieving a specific set of outcomes for Agile software development, such as the outcomes described in the USCIS policy. We will continue to work with USCIS to obtain additional documentation about actions it is taking to address this recommendation.
    Recommendation: To provide reasonable assurance that the program executes Agile software development for USCIS ELIS consistent with its own policies and guidance and follows applicable leading practices, the Secretary of DHS should direct the Director of USCIS to direct the USCIS CIO, in coordination with the DHS CIO and the Chief of OTC, to review and update, as needed, existing policies and guidance and consider additional controls to monitor program performance and report to appropriate entities through the collection of reliable metrics.

    Agency: Department of Homeland Security
    Status: Open

    Comments: As of July 2017, USCIS had taken steps to address this recommendation. For example, in May 2017, USCIS provided updated policy governing the development of software that called for teams to prepare an Operations Monitoring Plan or dashboard showing the practices, tools, and measures that will monitor applications in production. The agency also provided a series of documents from internal systems and processes intended to monitor performance, such as a product dashboard for analyzing code quality (i.e., SonarQube) and a report from its Independent Verification and Validation team. However, the program was undergoing a re-baseline and had yet to document updated cost, schedule, and performance expectations against which to monitor. Moreover, the agency did not demonstrate that other metrics, such as customer satisfaction and team velocity, were being reliably collected. We will continue to work with USCIS to obtain additional documentation about actions it is taking to address this recommendation.
    Recommendation: To help manage the USCIS ELIS system, the Secretary of DHS should direct the Director of USCIS to direct the USCIS CIO, in coordination with the DHS CIO and the Chief of OTC, to review and update existing policies and guidance and consider additional controls to conduct unit and integration, and functional acceptance tests, and code inspection consistent with stated program goals.

    Agency: Department of Homeland Security
    Status: Open

    Comments: As of July 2017, USCIS had taken steps to address this recommendation. For example, in May 2017, USCIS provided artifacts from internal systems in place to monitor software development performance. These metrics monitored aspects of testing, such as code quality and code coverage. However, the program did not provide an updated Test and Evaluation Master Plan, which is a document it will produce as part of its ongoing effort to re-baseline. A Test and Evaluation Master Plan sets the testing expectations for the program as agreed upon with its stakeholders in DHS and USCIS. The updated plan will provide a basis for further evaluation of the steps DHS and USCIS have taken to address this recommendation. Moreover, the agency did not demonstrate that functional acceptance tests were being conducted in accordance with stated program goals. For example, the agency did not provide acceptance criteria or the associated tests demonstrating that user stories passed the defined acceptance criteria. We will continue to work with USCIS to obtain additional documentation about actions it is taking to address this recommendation.
    Recommendation: To help manage the USCIS ELIS system, the Secretary of DHS should direct the Director of USCIS to direct the USCIS CIO, in coordination with the DHS CIO and the Chief of OTC, to review and update existing policies and guidance and consider additional controls to develop complete test plans and cases for interoperability and end user testing, as defined in the USCIS Transformation Program Test and Evaluation Master Plan, and document the results.

    Agency: Department of Homeland Security
    Status: Open

    Comments: As of July 2017, DHS and USCIS had not provided information demonstrating that they had addressed this recommendation. In October 2016, DHS provided a written response indicating that an internal process for revisiting the USCIS ELIS Test and Evaluation Master Plan had been initiated, with participation from all relevant stakeholder groups. A Test and Evaluation Master Plan sets the testing expectations for the program as agreed upon with its stakeholders in DHS and USCIS. The updated plan will provide a basis for further evaluation of the steps DHS and USCIS have taken to address this recommendation. The letter also stated that USCIS had begun to work on a policy for new interoperability test procedures. Moreover, the letter added that end user testing is a continuing activity, including providing feedback of observed issues into the development queue, with the slow launch of the naturalization capabilities in USCIS ELIS being a model. However, as of July 2017, DHS and USCIS had not provided new information about the status of this recommendation. We will continue to work with DHS and USCIS to obtain additional documentation about actions they are taking to address this recommendation.
    Director: Michele Mackin
    Phone: (202) 512-4841

    2 open recommendations
    Recommendation: To enhance DHS leadership's ongoing efforts to improve the affordability of the department's major acquisition portfolio, and to ensure adequate communication with Congress, the Secretary of the Department of Homeland Security should ensure that the fiscal year 2017 Future Years Homeland Security Program report, which DHS must submit to Congress at or about the same time as the President's fiscal year 2018 budget request, reflects the results of any tradeoffs stemming from the acquisition affordability reviews recommended above.

    Agency: Department of Homeland Security
    Status: Open

    Comments: In providing comments on this report, the Department of Homeland Security (DHS) concurred with this recommendation, and stated that the fiscal year 2017 Future Years Homeland Security Program (FYHSP) report would reflect decisions made in response to our second recommendation. DHS expected to release the FYHSP report shortly after the President's fiscal year 2018 budget request in May 2017. However, the transition to a new administration delayed the release of the FYHSP report. Once available, GAO will evaluate the FYHSP report to determine whether DHS has met the intent of this recommendation.
    Recommendation: To enhance DHS leadership's ongoing efforts to improve the affordability of the department's major acquisition portfolio, and to help ensure programs secure stable funding that matches resources to requirements, the Secretary of the Department of Homeland Security should require components to establish formal, repeatable processes for addressing major acquisition affordability issues, similar to the process the Transportation Security Administration has established.

    Agency: Department of Homeland Security
    Status: Open

    Comments: In providing comments on this report, the Department of Homeland Security (DHS) concurred with this recommendation, and stated that DHS headquarters would ensure all components are updating their cost estimates each year to inform the annual resource allocation process by March 31, 2017. However, DHS did not establish a requirement that components do so through formal, repeatable processes for addressing major acquisition affordability issues, similar to the process the Transportation Security Administration has established. As of August 2017, seven of DHS's components were in the process of establishing formal, repeatable processes for addressing affordability issues, but had not completed these efforts. GAO will continue to review the components' progress to determine whether the components' actions meet the intent of this recommendation.
    Director: Michele Mackin
    Phone: (202) 512-4841

    1 open recommendations
    Recommendation: To address different interpretations of cutter boat requirements, the Commandant of the Coast Guard should direct the NSC program office to clarify the NSC's key performance parameters for the cutter boat operations (specifically the launch and recovery of cutter boats).

    Agency: Department of Homeland Security: United States Coast Guard
    Status: Open

    Comments: The Coast Guard is in the process of updating the operator's handbook for the Long Range Interceptor II cutter boat to clarify that it is capable of operating through sea state 5, which will meet the National Security Cutter's key performance parameter related to cutter boat operations. According to Coast Guard officials, the updated operator's handbook should be signed and approved between August and November 2017.
    Director: Michele Mackin
    Phone: (202) 512-4841

    3 open recommendations
    including 1 priority recommendation
    Recommendation: To ensure that the Navy has provided a clear direction for the future of the program before committing funding to construct additional ships, Congress should consider, given the uncertainties over the long term about the ship's survivability and lethality and proposed changes to future ships, consider not fully funding the Navy's request for future LCS ships beyond fiscal year 2016, pending the completion and analysis of the final survivability assessments for both variants due in 2018.

    Agency: Congress
    Status: Open

    Comments: Although Congress did not take action on this Matter for Congressional Consideration in the fiscal year 2017 NDAA, we will continue to monitor this matter to see if Congress implements future restrictions prior to the final survivability assessments being completed. The Navy's final survivability assessment report is planned for fiscal year 2018, with DOT&E's assessment to follow.
    Recommendation: To ensure that the Navy has a sound acquisition approach moving forward, the Secretary of Defense should require the Navy to solicit an independent technical assessment from an organization like a ship classification society on the survivability of the Independence variant seaframe and its ability to meet its applicable requirements.

    Agency: Department of Defense
    Status: Open
    Priority recommendation

    Comments: DOD did not concur with our recommendation advocating an independent technical assessment by a classification society, stating that such an organization could not provide an independent look and was not technically competent to conduct such an evaluation. To fully implement this recommendation, we continue to believe that an independent assessment performed by the American Bureau of Shipping, or some other independent entity with relevant subject matter expertise would be valuable to understanding seaframe performance, which remains a significant uncertainty. Although the Navy has conducted rough water, ship shock, and total ship survivability testing, it has not demonstrated that the ship will achieve survivability requirements. Completion of the final survivability assessment report is anticipated in fiscal year 2018. Further, although the Navy has not completed its analytical reports of the rough water events, both Littoral Combat Ship variants sustained some damage, and the Director, Operational Test and Evaluation, has expressed concern with the testing and results from full ship shock trials and total ship survivability testing. An independent technical assessment of the Independence variant's survivability would help solidify the Navy's understanding of the ship's expected performance, and takes on added relevance given that the Navy's plans for a frigate award in fiscal year 2020 may include a downselect decision to a Littoral Combat Ship-based seaframe design. In July 2017, the LCS Program Office indicated that there is no additional information to provide, as the Navy maintains its non-concurrence with this recommendation. We will continue monitoring this recommendation to see if the Navy solicits such an evaluation, given continuing concerns about the ships survivability from the Director of Operational Test and Evaluation.
    Recommendation: To ensure that the program has requirements that are testable and measurable and to improve realism of LCS operational testing, the Secretary of Defense should direct the Secretary of the Navy to investigate resourcing and conducting more operationally stressing SUW mission package testing onboard LCS, to include testing in a clutter environment and diverse weather and tactical scenarios to help ensure that the ships can operate effectively in their intended environment.

    Agency: Department of Defense
    Status: Open

    Comments: DOD partially concurred with GAO's recommendation, stating it will provide sufficient test resources but does not believe that testing 'every aspect' of weather and tactics is necessary. The Navy has since received DOT&E approval of a partial update to the LCS Test and Evaluation Master Plan (TEMP), which was required before March 2016 by a restriction included in the NDAA for Fiscal Year 2016. According to the department, the update included changes to the scope of SUW mission package testing. The LCS Program Office stated that a full LCS TEMP is expected to be approved by the Office of the Secretary of Defense in the last quarter of fiscal year 2017. OSD approval of the TEMP would likely meet the intent of our recommendation; however, until it is approved, we will monitor the TEMP status and keep this recommendation open.
    Director: Michele Mackin
    Phone: (202) 512-4841

    1 open recommendations
    Recommendation: To help ensure that actions taken to improve the test and evaluation process address identified challenges, the Administrator of TSA should finalize all aspects of the third party testing strategy before implementing further third party testing requirements for vendors to enter testing.

    Agency: Department of Homeland Security: Transportation Security Administration
    Status: Open

    Comments: In its response to this recommendation, the Department of Homeland Security concurred and identified initial planned actions to implement the finalize third party strategy. Subsequently, in the Spring of 2016, the TSA Office of Security Capabilities finalized its third party tester application and approval process; established quality conformance standards for potential third party testers; and gathered and considered industry feedback on potential third party test strategy consequences among other actions. Collectively, TSA established and published program requirements and procedures for the third party test strategy. In late 2016, TSA formally delayed its planned implementation of the third-party testing program by a calendar year to now be completed by December 31, 2017. TSA cited a need to conduct additional assessments, coordination challenges, and larger TSA security equipment related initiatives as the reasons for the delay. As part of its regular recommendation status reporting to GAO, TSA in the Spring 2017, noted that it is on track to meet the intent of the recommendation by the later revised date. TSA noted it had recently updated its qualification process by which qualified product lists will be populated and has already incorporated various aspects of third party testing for legacy security equipment qualification.
    Director: Cristina Chaplain
    Phone: (202) 512-4841

    4 open recommendations
    Recommendation: To better position DOD as it continues pursuing GPS modernization, to have the information necessary to make decisions on how best to improve that modernization, and to mitigate risks to sustaining the GPS constellation, the Secretary of Defense should convene an independent task force comprising experts from other military services and defense agencies with substantial knowledge and expertise to provide an assessment to the Under Secretary of Defense for Acquisition, Technology, and Logistics of the OCX program and concrete guidance for addressing the OCX program's underlying problems, particularly including: (1) A detailed engineering assessment of OCX defects to determine the systemic root causes of the defects; (2) Whether the contractor's software development procedures and practices match the levels described in the OCX systems engineering and software development plans; and (3) Whether the contractor is capable of executing the program as currently resourced and structured.

    Agency: Department of Defense
    Status: Open

    Comments: DOD concurred with this recommendation. Prior to the program declaring a Nunn-McCurdy breach on June 30, 2016, the only independent assessment was conducted by Defense Digital Services and was limited in focus to software development. Air Force notes a completion date of independent assessment on Sept 29, 2017. Once received, we will evaluate whether that meets the intent of the recommendation.
    Recommendation: To better position DOD as it continues pursuing GPS modernization, to have the information necessary to make decisions on how best to improve that modernization, and to mitigate risks to sustaining the GPS constellation, the Secretary of Defense should develop high confidence OCX cost and schedule estimates based on actual track record for productivity and learning curves.

    Agency: Department of Defense
    Status: Open

    Comments: DOD concurred with this recommendation. Prior to the program declaring a Nunn-McCurdy breach on June 30, 2016, no high confidence cost assessment was completed. The Air Force and contractor provided schedule assessments that were not evaluated and considered low-risk, but were directed to execute a 24 month schedule extension with no assessment of its feasibility and that did not take into account past contractor performance. Pending Nunn-McCurdy documentation and repeat of Milestone B, there is no evidence a high confidence cost or schedule has been put in place. Once we receive documentation on approval of Milestone B, we will reevaluate.
    Recommendation: To better position DOD as it continues pursuing GPS modernization, to have the information necessary to make decisions on how best to improve that modernization, and to mitigate risks to sustaining the GPS constellation, the Secretary of Defense should direct the Air Force to retain experts from the independent task force as a management advisory team to assist the OCX program office in conducting regular systemic analysis of defects and to help ensure OCX corrective measures are implemented successfully and sustained.

    Agency: Department of Defense
    Status: Open

    Comments: DOD concurred with this recommendation. Prior to the program declaring a Nunn-McCurdy breach on June 30, 2016, Defense Digital Services were initially retained for a month and subsequently remain embedded with contractor software developers to provide advice on development and process improvements. Upon completion of the Nunn-McCurdy review and continued involvement of Defense Digital Services, we will examine the extent to which the program has met this recommendation if the program is recertified to determine if this recommendation was met. Air Force did not provide an update to this recommendation in 2017, but program still has not had Milestone B approved and the Defense Digital Services group is no longer engaged on OCX.
    Recommendation: To better position DOD as it continues pursuing GPS modernization, to have the information necessary to make decisions on how best to improve that modernization, and to mitigate risks to sustaining the GPS constellation, the Secretary of Defense should put in place a mechanism for ensuring that the knowledge gained from the OCX assessment is used to determine whether further programmatic changes are needed to strengthen oversight.

    Agency: Department of Defense
    Status: Open

    Comments: DOD concurred with this recommendation. Senior quarterly reviews continue of the OCX program and have been in place since December 2015. Documentation still pending on Milestone B to see if these reviews have informed programmatic changes that better position DOD to complete this acquisition.
    Director: Susan Fleming
    Phone: (202) 512-2834

    1 open recommendations
    Recommendation: To improve FMCSA's oversight of state's CDL testing programs and ensure the FMCSA's oversight activities provide reasonable assurance that states' programs comply with federal regulations, the Secretary of Transportation should direct the FMCSA Administrator to improve or obtain a mechanism for tracking oversight activities in order that FMCSA management has a clear and accurate understanding of oversight activities and that they are being conducted in accordance with agency oversight policy.

    Agency: Department of Transportation
    Status: Open

    Comments: The Department of Transportation concurred with GAO's recommendation and is in the process of developing a new system for tracking oversight activities and results.
    Director: Cristina Chaplain
    Phone: (202) 512-4841

    2 open recommendations
    Recommendation: To strengthen MDA's acquisition efforts and help support oversight and to ensure MDA makes sound investment decisions on improving homeland ballistic missile defense, the Secretary of Defense should direct MDA to make the department's analysis of alternatives an integral part of its planning effort and delay any decisions to begin development of the new GMD Redesigned Kill Vehicle until: a) the department's analysis of alternatives is completed and identifies the best solution to pursue; and b) Congressional and DOD decision makers have been provided the results of that analysis.

    Agency: Department of Defense
    Status: Open

    Comments: DOD partially concurred with our recommendation. As of July 2017, MDA has yet to finalize the analysis of alternatives (AOA) report. MDA officials stated the results of the analysis from the AOA were reached in early fiscal year 2015 and informed the decision to pursue the RKV. The AOA results were summarized and briefed to staff members from the congressional defense committees and senior DOD leadership. MDA subsequently submitted the report to DOD's Office of Cost Assessment and Program Evaluation (CAPE) for a sufficiency review but an official from the Office of the Secretary of Defense stated in July 2017 that CAPE has yet to perform the review. We await the final AOA report to make a determination of whether the recommendation was implemented.
    Recommendation: To strengthen MDA's acquisition efforts and help support oversight, drawing from information it already has, the Secretary of Defense should direct MDA to report annually to Congress its plans for, and achieved progress in developing and delivering integrated Ballistic Missile Defense System (BMDS)-level capabilities. This reporting should include: a) planned integrated BMDS-level capabilities, including dates for when capability is planned for delivery; and b) element-level upgrades needed for delivery of the integrated BMDS capability, including dates that these upgrades need to be available for integration into the BMDS capability.

    Agency: Department of Defense
    Status: Open

    Comments: In providing comments on this report, the Department of Defense concurred with our recommendation to annually report to Congress its plans and progress made in delivering integrated BMDS-level capabilities, including delivery dates for the capabilities and element upgrades needed to support these capacities. Previously, MDA planned to provide Congress a table summarizing the development and delivery of integrated BMDS capabilities annually, beginning in the 2016 BMDS Accountability Report (BAR). We reviewed the 2016 BAR and determined the report did not include the full extent of information regarding plans and progress made in delivery integrating BMDS-level capabilities described in our recommendation. To remain consistent with the methodology for our annual missile defense review, we will assess the 2017 BAR in next year's review to determine if it includes this information.
    Director: Michele Mackin
    Phone: (202) 512-4841

    2 open recommendations
    Recommendation: To improve DHS's management of major acquisition programs, the Secretary of Homeland Security should ensure future baselines for all of TSA's major acquisition programs capture the overall historical record of change.

    Agency: Department of Homeland Security
    Status: Open

    Comments: In providing comments on this report, the Department of Homeland Security (DHS) concurred with this recommendation, and stated that the Transportation Security Administration (TSA) will begin to incorporate an addendum to future Acquisition Program Baselines (APB) that will provide a single source to show the changes to cost, schedule, and performance metrics, beginning with the initial program baseline and showing traceability of all interim approved versions to the current APB. DHS estimated it would complete this effort April 30, 2016. As of August 2017, DHS leadership had approved updated versions of the two APBs that were the basis for this recommendation. Both included addendums with metrics from prior APBs, but raised questions about traceability to the current cost, schedule, and performance metrics. GAO will assess the updated APBs as a part of its annual review of select DHS major acquisition programs to determine whether the department has addressed the recommendation.
    Recommendation: To more accurately communicate DHS's funding plans for USCG's major acquisition programs, the Secretary of Homeland Security should ensure the funding plans presented to Congress in fiscal year 2015 are comprehensive and clearly account for all operations and maintenance funding DHS plans to allocate to each of the USCG's major acquisition programs.

    Agency: Department of Homeland Security
    Status: Open

    Comments: In providing comments on this report, the Department of Homeland Security (DHS) concurred with this recommendation, and stated that the U.S. Coast Guard and the DHS Chief Financial Officer will develop a plan to address this recommendation by September 30, 2015, then work together to fully implement the plan. DHS estimated it would complete this effort March 31, 2016. However, the USCG encountered technical challenges during this process and was unable to implement the plan by that time. The U.S. Coast Guard has revised the estimated completion date, and now anticipates it will be able to address this recommendation in fiscal year 2020.
    Director: Rebecca Gambler
    Phone: (202) 512-8777

    6 open recommendations
    Recommendation: To ensure that CBP's land mobile radio systems are functioning as intended in each location and are meeting user needs, the CBP Commissioner should develop a plan to monitor the performance of its deployed radio systems.

    Agency: Department of Homeland Security: United States Customs and Border Protection
    Status: Open

    Comments: When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.
    Recommendation: To ensure the ICE TACCOM program is effectively managed, the Assistant Secretary of ICE should develop a program plan to ensure that the agency establishes the appropriate documentation of resource needs, program goals, and measures to monitor the performance of its deployed radio systems.

    Agency: Department of Homeland Security: Directorate of Border and Transportation Security: Bureau of Immigration and Customs Enforcement
    Status: Open

    Comments: When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.
    Recommendation: To improve CBP training efforts, CBP Commissioner should develop and implement a plan to address any skills gaps for CBP agents and officers related to understanding the new digital radio systems and interagency radio use protocols.

    Agency: Department of Homeland Security: United States Customs and Border Protection
    Status: Open

    Comments: When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.
    Recommendation: To improve CBP training efforts, CBP Commissioner should develop a mechanism to verify that all Border Patrol and OFO radio users receive radio training.

    Agency: Department of Homeland Security: United States Customs and Border Protection
    Status: Open

    Comments: When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.
    Recommendation: To improve ICE training efforts, the Assistant Secretary of ICE should develop and implement a plan to address any skills gaps for ICE agents related to understanding the new digital radio systems and interagency radio use protocols.

    Agency: Department of Homeland Security: Directorate of Border and Transportation Security: Bureau of Immigration and Customs Enforcement
    Status: Open

    Comments: When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.
    Recommendation: To improve ICE training efforts, the Assistant Secretary of ICE should develop a mechanism to verify that all ICE radio users receive radio training.

    Agency: Department of Homeland Security: Directorate of Border and Transportation Security: Bureau of Immigration and Customs Enforcement
    Status: Open

    Comments: When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.
    Director: Jennifer A. Grover
    Phone: (202) 512-7141

    7 open recommendations
    including 4 priority recommendations
    Recommendation: To ensure that the NSCs can be operated and maintained in the most demanding environments based on mission and maintenance requirements prior to implementation of the CRC, the Coast Guard should, as expeditiously as possible under its capacity limits and fiscal constraints, fulfill the staffing requirements recommended in the 2011 manpower requirements analysis, including ensuring that while implementing the interim 210 Plan, the NSCs operate with sufficient numbers of crew members who possess the recommended mix of skills and abilities.

    Agency: Department of Homeland Security: United States Coast Guard
    Status: Open

    Comments: In August 2015, the Coast Guard stated that funding to fulfill the staffing requirements recommended in the 2011 manpower requirements analysis for the National Security Cutter is under review and is expected to be determined by February 2016. On January 24, 2017, the Coast Guard noted that it expects the crew increase to be reflected in the FY 2017 Appropriation.
    Recommendation: To improve the Coast Guard's ability to make informed decisions about the overall feasibility of its goal to achieve 230 DAFHP using the CRC, and to ensure the effectiveness of the scheduled CRC feasibility test, the Coast Guard's CRC plan, scheduled to be completed by December 2017, should specify mitigation actions to effectively address the risk factors identified in this report, including determining the appropriate number of NSC crew and shoreside-based support personnel with the right mix of skills and abilities and having them in place when the Coast Guard tests the CRC.

    Agency: Department of Homeland Security: United States Coast Guard
    Status: Open
    Priority recommendation

    Comments: In July 2015, the Coast Guard provided documentation of a manpower requirements analysis and manpower requirements determination, which specify the number of shoreside-based support personnel, such as engineering and maintenance, that are needed to support 4 National Security Cutters (NSC) based in Alameda, CA (i.e., three of the four NSCs using the crew rotational concept). In March 2016, the Coast Guard stated that it would request resources, as appropriate, to ensure that these personnel are in place prior to testing and expects to close this recommendation by November 2017. On January 24, 2017, the Coast Guard noted that it is continuing to develop the final test plan.
    Recommendation: To improve the Coast Guard's ability to make informed decisions about the overall feasibility of its goal to achieve 230 DAFHP using the CRC, and to ensure the effectiveness of the scheduled CRC feasibility test, the Coast Guard's CRC plan, scheduled to be completed by December 2017, should specify mitigation actions to effectively address the risk factors identified in this report, including addressing the misalignment of the crewing concept to be used in the planned CRC test, as compared to the NSC homeporting plan, so that the CRC test is conducted in an operationally realistic environment and that the test results can be used to determine the optimal schedules for rotating crews and performing maintenance.

    Agency: Department of Homeland Security: United States Coast Guard
    Status: Open
    Priority recommendation

    Comments: In March 2016, the Coast Guard stated it continues to analyze and develop various testing plans and will submit a final plan to increase the NSCs' days away from homeport to Congress by December 2017. On January 24, 2017, the Coast Guard noted that it continues to analyze and develop various testing plans to reach 225 DAFHP onboard NSCs. The test will be performed by Coast Guard Pacific Area during normal operations over an extended period to ensure it is operationally realistic; lessons learned will be used to inform future crewing models. The Coast Guard will submit a final test plan to Congress prior to the deadline set forth by the 2012 Coast Guard Authorization Bill (end of 2017).
    Recommendation: To improve the Coast Guard's ability to make informed decisions about the overall feasibility of its goal to achieve 230 DAFHP using the CRC, and to ensure the effectiveness of the scheduled CRC feasibility test, the Coast Guard's CRC plan, scheduled to be completed by December 2017, should specify mitigation actions to effectively address the risk factors identified in this report, including addressing the potential impacts of wide variations between alternative CRC deployment schedules.

    Agency: Department of Homeland Security: United States Coast Guard
    Status: Open
    Priority recommendation

    Comments: In March 2016, the Coast Guard stated it continues to analyze and develop various testing plans and will submit a final plan to increase the NSCs' days away from homeport to Congress by December 2017. On January 24, 2017, the Coast Guard noted that it continues to analyze and develop various testing plans to reach 225 DAFHP onboard NSCs. The test will be performed by Coast Guard Pacific Area during normal operations over an extended period to ensure it is operationally realistic; lessons learned will be used to inform future crewing models. The Coast Guard will submit a final test plan to Congress prior to the deadline (end of 2017) set forth by the 2012 Coast Guard Authorization Bill.
    Recommendation: To improve the Coast Guard's ability to make informed decisions about the overall feasibility of its goal to achieve 230 DAFHP using the CRC, and to ensure the effectiveness of the scheduled CRC feasibility test, the Coast Guard's CRC plan, scheduled to be completed by December 2017, should specify mitigation actions to effectively address the risk factors identified in this report, including expanding the Coast Guard's training infrastructure capacity to provide crew members with the necessary training for off-cycle rotating NSC crew members under the CRC.

    Agency: Department of Homeland Security: United States Coast Guard
    Status: Open
    Priority recommendation

    Comments: In March 2016, the Coast Guard stated it continues to analyze and develop various testing plans and will submit a final plan to increase the NSCs' days away from homeport to Congress by December 2017. On January 24, 2017, the Coast Guard noted that it continues to analyze and develop various testing plans to reach 225 DAFHP onboard NSCs. The test will be performed by Coast Guard Pacific Area during normal operations over an extended period to ensure it is operationally realistic; lessons learned will be used to inform future crewing models. The Coast Guard will submit a final test plan to Congress prior to the deadline set forth by the 2012 Coast Guard Authorization Bill (the end of 2017).
    Recommendation: To ensure that the Coast Guard is making progress in a timely manner to address and effectively mitigate the risk factors identified above, the Coast Guard should develop interim milestones for the various actions to be taken on each of the risk factors as the Coast Guard completes the CRC Plan.

    Agency: Department of Homeland Security: United States Coast Guard
    Status: Open

    Comments: In March 2016, the Coast Guard stated it continues to analyze and develop various testing plans and will submit a final plan to increase the NSCs' days away from homeport to Congress by December 2017. On January 24, 2017, the Coast Guard noted that the analysis is ongoing, and will be addressed in the test plan submitted to Congress in accordance with the 2012 Coast Guard Authorization Bill (end of 2017).
    Recommendation: Finally, to ensure that the Coast Guard is making progress in developing alternative measures that provide more accurate indicators of operational performance in a timely manner, the Coast Guard should establish time frames and interim milestones for developing and implementing these alternative measures for use prior to CRC testing. These measures could then be used for both the NSCs, as well as for other cutters, such as the Offshore Patrol Cutter, that currently use or plan to use the traditional DAFHP performance measure.

    Agency: Department of Homeland Security: United States Coast Guard
    Status: Open

    Comments: In August 2015, the Coast Guard stated that its analysis of alternative measures for use prior to testing the NSCs to use the crew rotational concept was ongoing and did not provide a date for completion. On January 24, 2017, the Coast Guard noted that, in the process of analyzing an alternative to DAFHP, it found that while its enterprise data management system collects the necessary data; the system didn't have an efficient way of aggregating the data for analysis. The first step to develop an appropriate enterprise measure is to improve the data management system through a software change. Once the change is complete, the system will be evaluated and tested to ensure that an aggregated report of discrete Coast Guard Cutter activity is accurate and reliable. The change will include the functionality to separate underway from in-port activity. After examining the results from the software change, the Coast Guard will evaluate an alternative to DAFHP. However, all analyses to date indicate DAFHP remains an important measure for personnel operations tempo and cutter scheduling and will not be eliminated as an available measure. The Coast Guard updated the estimated completion date for this recommendation to July 2018.
    Director: Dave Powner
    Phone: (202) 512-9286

    1 open recommendations
    Recommendation: To address risks in the GOES-R program development and to help ensure that the satellite is launched on time, the Secretary of Commerce should direct the NOAA Administrator to address shortfalls in defect management identified in this report, including the lack of clear guidance on defect definitions, what defect metrics should be collected and reported, and how to establish a defect's priority or severity.

    Agency: Department of Commerce
    Status: Open

    Comments: NOAA agreed with this recommendation. The agency subsequently reported that contractors are required to report defects and that the agency can place a hold or put liens against contractors if defects are not addressed. NOAA also provided documentation to support its recurring meetings at which defects are addressed. Additionally, NOAA provided documentation on its defect reporting requirements and definitions. However, NOAA did not provide documentation showing what defect metrics should be collected and reported, and how to establish a defect's priority or severity. We will continue to monitor the agency's actions on this recommendation.
    Director: Michele Mackin
    Phone: (202) 512-4841

    1 open recommendations
    Recommendation: The legislated cost cap for Ford-class aircraft carrier construction provides a limit on procurement funds. However, the legislation also provides for adjustments to the cost cap. To understand the true cost of each Ford-class ship, Congress should consider revising the cost cap legislation to ensure that all work included in the initial ship cost estimate that is deferred to post-delivery and outfitting account is counted against the cost cap. If warranted, the Navy would be required to seek statutory authority to increase the cap.

    Agency: Congress
    Status: Open

    Comments: This recommendation remains open to allow Congress time to consider legislation amending the cost cap for the Ford class of aircraft carriers. The current version of the fiscal year 2018 National Defense Authorization Act (H.R. 2810) does not amend the current cost cap legislation.
    Director: Cary Russell
    Phone: (202) 512-5431

    6 open recommendations
    including 1 priority recommendation
    Recommendation: To help DOD develop an affordable sustainment strategy for the F-35, the Secretary of Defense should direct the Under Secretary of Defense for Acquisitions, Technology and Logistics to direct the F-35 Program Executive Officer to establish affordability constraints linked to, and informed by, military service budgets that will help guide sustainment decisions, prioritize requirements, and identify additional areas for savings by March 2015, at which point the Future Support Construct decision will be approved.

    Agency: Department of Defense
    Status: Open
    Priority recommendation

    Comments: DOD concurred with our recommendation and stated in April 2017 that the F-35 Program Executive Officer and the F-35 enterprise have expanded their collaborative effort to reduce F-35 operating and support (O&S) costs to ensure that they deliver affordable readiness for the F-35 fleet. In an effort to reduce overall O&S costs, the department has undertaken several initiatives. For example, according to DOD, as of January 2017, a program office "cost war room" initiative had reduced the 2012 F-35 annual cost estimate by $60.7 billion. Additionally, according to DOD, a Reliability and Maintainability Improvement Program has resulted in a $1.7 billion O&S cost avoidance through the program's life cycle. Other efforts are also under way that aim to help reduce O&S costs by better informing sustainment decision-making. While the department is taking steps to try to reduce overall O&S costs, the program has yet to develop affordability constraints linked to the military services' budgets. Without affordability constraints that are linked to military service budgets, it remains unclear the extent to which the military services can afford to operate and sustain the F-35 throughout its life cycle as currently planned.
    Recommendation: To help DOD address key risks to F-35 affordability and operational readiness, and to improve the reliability of its O&S cost estimates for the life cycle of the program, the Secretary of Defense should direct the F-35 Program Executive Officer, to enable DOD to better identify, address, and mitigate performance issues with the Autonomic Logistics Information System (ALIS) that could have an effect on affordability, as well as readiness, to establish a performance-measurement process for ALIS that includes, but is not limited to, performance metrics and targets that (1) are based on intended behavior of the system in actual operations and (2) tie system performance to user requirements.

    Agency: Department of Defense
    Status: Open

    Comments: According to DOD officials, the ALIS Integrated Product Team (IPT) is continuing to work with the Joint Program Office's Performance Based Logistics (PBL) team to further develop and refine appropriate metrics for inclusion into future sustainment contracts. Although DOD has made progress in developing performance metrics for ALIS, as of September 2017, DOD has yet to develop metrics that are based on intended behavior of the system and tie system performance to user requirements. Until this progression is made, this recommendation will remain open.
    Recommendation: To help DOD address key risks to F-35 affordability and operational readiness, and to improve the reliability of its O&S cost estimates for the life cycle of the program, the Secretary of Defense should direct the F-35 Program Executive Officer, to develop a high level of confidence that the aircraft will achieve its R+M goals, to develop a software reliability and maintainability (R+M) assessment process, with metrics, by which the program can monitor and determine the effect that software issues may have on overall F-35 R+M issues.

    Agency: Department of Defense
    Status: Open

    Comments: DOD has an R&M assessment process in place, but as of September 2017, had not developed a process that would focus directly on software reliability and maintainability. Until DOD develops a process more focused on software and its effects on overall R&M issues, this recommendation will remain open.
    Recommendation: To help DOD address key risks to F-35 affordability and operational readiness, and to improve the reliability of its O&S cost estimates for the life cycle of the program, the Secretary of Defense should direct the F-35 Program Executive Officer, to promote competition, address affordability, and inform its overarching sustainment strategy, to develop a long-term Intellectual Property (IP) Strategy to include, but not be limited to, the identification of (1) current levels of technical data rights ownership by the federal government and (2) all critical technical data needs and their associated costs.

    Agency: Department of Defense
    Status: Open

    Comments: DOD has still not developed an overall strategy that would identify data rights ownership, needs, and costs. As of September 2017, the program had taken some steps to develop an Intellectual Property Strategy, but has not identified all critical needs and their associated costs. Program office officials said that they are currently working with the prime contractor to develop a list of technical data requirements. Until this strategy is developed, this recommendation will remain open.
    Recommendation: To help DOD address key risks to F-35 affordability and operational readiness, and to improve the reliability of its O&S cost estimates for the life cycle of the program, the Secretary of Defense should direct the F-35 Program Executive Officer, to understand the potential range of costs associated with the JPO F-35 O&S cost estimate, to conduct uncertainty analyses on future JPO estimates.

    Agency: Department of Defense
    Status: Open

    Comments: As of September 2017, DOD had not applied risk/uncertainty analyses to its cost estimates. Until it does so, this recommendation will remain open.
    Recommendation: To improve the reliability of the CAPE F-35 O&S cost estimate, the Secretary of Defense should direct the Director of CAPE, for future F-35 O&S cost estimates, to conduct uncertainty analyses to understand the potential range of costs associated with its estimates to reflect the most likely costs associated with the program.

    Agency: Department of Defense
    Status: Open

    Comments: According to DOD officials, the Cost Assessment and Program Evaluation (CAPE) has not updated its F-35 estimate subsequent to the release of GAO-14-778. Pending a major program change, CAPE will update the F-35 O&S estimate for the full-rate production decision point in the second quarter of fiscal year 2019. Until CAPE updates its F-35 estimate, we will not be able to determine if they will perform any uncertainty analyses on its cost estimate; therefore, this recommendation will remain open as of September 1, 2017.
    Director: Michele Mackin
    Phone: (202) 512-4841

    1 open recommendations
    including 1 priority recommendation
    Recommendation: The Under Secretary of Defense for Acquisition, Technology, and Logistics should require--before approving the release of the request for proposals for future contracts for either seaframe variant--that both variants: a. Have deployed to a forward overseas location; b. Have completed rough water, ship shock, and total ship survivability testing; and c. Have completed initial operational test and evaluation of the SUW mission package on the Freedom variant and the MCM mission package on the Independence variant.

    Agency: Department of Defense: Office of the Secretary of Defense: Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics
    Status: Open
    Priority recommendation

    Comments: DOD partially concurred with our recommendation, stating that it has every intention of completing as many as possible of the test and demonstration items that we identified in our recommendation before releasing the request for proposals (RFP) for future seaframe contracts, but disagreed that the release of the RFP should hinge on completion of these events. DOD officials stated that creating a break in the production of the seaframes would increase program costs and have significant industrial base considerations. To fully implement this recommendation, DOD should ensure that the Navy is procuring Littoral Combat Ships that meet its needs and that it does not continue to commit to additional ships until it demonstrates that it has attained some level of knowledge in key areas, such as ship survivability. The Navy has made progress since we made this recommendation, deploying both variants overseas and completing total ship survivability trials and full ship shock trials (FSST), as well as testing in rough water conditions. The LCS program stated that the results from rough water testing and shock trials are planned to be completed in the fourth quarter of fiscal year 2017. Still, the Navy has continued to award additional contracts for LCS before having demonstrated survivability capabilities, with some surface warfare package operational testing yet to be completed and mine countermeasures package initial operational capability delayed until 2020. This recommendation will remain open to allow for future Navy analysis and department action on this subject.
    Director: Michele Mackin
    Phone: (202) 512-4841

    2 open recommendations
    Recommendation: To help ensure that it receives accurate information on the full effect of funding decisions on acquisition programs, Congress should consider amending the law that governs the 5-year Capital Investment Plan to require the Coast Guard to submit cost and schedule information that reflects the impact of the annual President's budget request on each acquisition across the portfolio--in addition to the current practice of reporting the cost and schedule estimates in current program baselines.

    Agency: Congress
    Status: Open

    Comments: Thus far no congressional action has been taken on this Matter. We will continue to follow up with relevant congressional committees.
    Recommendation: To help the Coast Guard improve the long-term outlook of its portfolio, the Commandant of the Coast Guard should develop a 20-year fleet modernization plan that identifies all acquisitions needed to maintain the current level of service and the fiscal resources necessary to build the identified assets. The plan should also consider trade-offs if the fiscal resources needed to execute the plan are not consistent with annual budgets.

    Agency: Department of Homeland Security: United States Coast Guard
    Status: Open

    Comments: Based on this recommendation, Congress has requested that the Coast Guard develop a 20-year plan that identifies all acquisitions needed to maintain the Coast Guard's current level of service and the financial commitment necessary to achieve this plan. As a part of a series of testimonies in June and July 2017, we found that Coast Guard officials stated they are developing a 20-year Capital Investment Plan (CIP), but the timeframe for completion is unknown. The Coast Guard does, however, submit a 5-year CIP annually to Congress that projects acquisition funding needs for the upcoming 5 years. GAO found the CIPs do not match budget realities in that tradeoffs are not included. In the 20-year CIP, GAO would expect to see all acquisitions needed to maintain current service levels and the fiscal resources to build the identified assets as well as tradeoffs in light of funding constraints.
    Director: Cristina Chaplain
    Phone: (202) 512-4841

    2 open recommendations
    including 2 priority recommendations
    Recommendation: To provide the Congress with the necessary insight into program affordability, ensure its ability to effectively monitor total program costs and execution, and to facilitate investment decisions, the NASA's Administrator should direct the Human Exploration and Operations Mission Directorate to establish a separate cost and schedule baseline for work required to support the SLS Block I Exploration Mission-2 (EM-2) and report this information to the Congress through NASA's annual budget submission. If NASA decides to fly the SLS Block I beyond EM-2, establish separate life cycle cost and schedule baseline estimates for those efforts, to include funding for operations and sustainment, and report this information annually to Congress via the agency's budget submission.

    Agency: National Aeronautics and Space Administration
    Status: Open
    Priority recommendation

    Comments: NASA partially agreed with this recommendation, stating that it defined and documented life cycle costs for SLS to a first demonstrated capability, consistent with cost estimating best practices and NASA project and program management policy and that it would report costs associated with the second exploration mission via its annual budget submission. Best practices for cost estimating recognize that NASA's evolutionary development approach for SLS helps reduce risk and provide capabilities more quickly, but reporting costs via the budget alone will not provide information about potential costs over the long-term and progress cannot be assessed without a baseline that serves as a means to compare current costs against expected costs. To address this recommendation, NASA needs to establish separate cost and schedule baselines for work required to support SLS for EM-2.
    Recommendation: To provide the Congress with the necessary insight into program affordability, ensure its ability to effectively monitor total program costs and execution, and to facilitate investment decisions, because NASA intends to use the increased capabilities of the SLS, Orion, and Ground Systems Development and Operations efforts well into the future and has chosen to estimate costs associated with achieving the capabilities, the NASA's Administrator should direct the Human Exploration and Operations Mission Directorate to establish separate cost and schedule baselines for each additional capability that encompass all life cycle costs, to include operations and sustainment. When NASA cannot fully specify costs due to lack of well-defined missions or flight manifests, forecast a cost estimate range -- including life cycle costs -- having minimum and maximum boundaries. These baselines or ranges should be reported to Congress annually via the agency's budget submission.

    Agency: National Aeronautics and Space Administration
    Status: Open
    Priority recommendation

    Comments: NASA partially agreed with this recommendation, stating that it had established separate programs for Space Launch System, Orion, and the ground systems and adopted a block upgrade approach for SLS. While NASA's prior establishment of SLS, Orion, and the ground systems as separate programs lends some insight into expected costs and schedule at the broader program level, it does not meet the intent of the recommendation because cost and schedule identified at that level is unlikely to provide the detail necessary to monitor the progress of each block against a baseline. To address this recommendation, NASA needs to establish separate cost and schedule baselines for each additional SLS, Orion, and Ground Systems Development and Operations capability blocks that encompass all life-cycle costs, to include operations and sustainment.
    Director: Cristina Chaplain
    Phone: (202) 512-4841

    2 open recommendations
    Recommendation: To strengthen MDA's acquisitions and help support oversight and, to the extent that MDA determines hardware or software modifications are required to address the September 2013 Aegis BMD SM-3 Block IB failure, the Secretary of Defense should direct, (a) the Director of the MDA to verify the changes work as intended through subsequent flight testing, and (b) the Under Secretary of Defense, Acquisitions, Technology, and Logistics to delay the decision to approve the program's full production until such testing demonstrates that the redesigned missile is effective and suitable.

    Agency: Department of Defense
    Status: Open

    Comments: In providing comments on this report, the Department of Defense partially concurred with our recommendation to delay full production decision until flight testing the Standard Missile-3 Block IB (SM-3 Block IB) with modifications that resulted from a September 2013 test failure. MDA recently completed 2 non-intercept flight tests to assess the redesign, these were non-interceptor tests and thus the extent to which the tests stressed the redesign and how well the redesigned component responded is still not fully understood. The assessment is currently ongoing and Full Production is planned for Fiscal Year 2018, after the assessment is completed.
    Recommendation: To strengthen MDA's acquisitions and help support oversight, and to demonstrate the Capability Enhancement-I's (CE-I) effectiveness against a longer range target in more challenging conditions and to confirm the design changes implemented to improve performance, as well as any changes needed to resolve the July 2013 CE-I flight test failure work as intended, the Secretary of Defense should direct MDA's Director to conduct a flight test of the CE-I interceptor once the cause of the failure has been determined and any mitigations have been developed.

    Agency: Department of Defense
    Status: Open

    Comments: DOD did not concur with the recommendation. In its response to the recommendation, DOD stated the decision to flight test a CE-I interceptor will be made by the Director, MDA, based on the judgment of stakeholders from the Office of the Secretary of Defense and combatant commanders on the need to perform a test. This year, MDA has indicated no change in direction regarding plans to conduct another CE-I flight test in the near term. The next CE-I flight test is not planned to occur until Fiscal Year 2018. That test is not a re-test of FTG-07; rather, it is a salvo test. It is unclear whether MDA will remain committed to including the CE-I in the test, as MDA has previously indicated it may use a different type of interceptor for that test. In addition, MDA only implemented a partial mitigation to address the FTG-07 failure cause and as such, MDA is not implementing all necessary corrections.
    Director: Thomas Melito
    Phone: (202) 512-9601

    4 open recommendations
    Recommendation: To improve the efficiency and accountability of the emergency food aid procurement process, the Secretary of Agriculture and Administrator of USAID should direct their staffs to work together to take steps to improve USDA's ability to account for U.S. government funds by ensuring that USAID provides USDA with accurate prepositioned commodity inventory data that USDA can independently verify.

    Agency: Department of Agriculture
    Status: Open

    Comments: As of May 2017, according to USDA officials, they are aware that USAID is working on a Statement of Work for a system to track prepositioned commodity inventory data.
    Recommendation: To improve the efficiency and accountability of the emergency food aid procurement process, the Secretary of Agriculture and Administrator of USAID should direct their staffs to work together to take steps to improve USDA's ability to account for U.S. government funds by ensuring that USAID provides USDA with accurate prepositioned commodity inventory data that USDA can independently verify.

    Agency: United States Agency for International Development
    Status: Open

    Comments: As of May 2017, USAID plans to have a contract to develop a system to track prepositioned commodity inventory data, by the end of fiscal year 2017.
    Recommendation: To improve the efficiency and accountability of the emergency food aid procurement process, the Secretary of Agriculture and Administrator of USAID should direct their staffs to work together to take steps to assess WBSCM's functionality by testing the international procurement functions that have been modified since April 2011 and documenting the results.

    Agency: Department of Agriculture
    Status: Open

    Comments: As of May 2017, USDA has held multiple meetings with USAID as part of its Business Management Improvement initiative, to assess Web Based Supply Chain Management's (WBSCM) functionality and test the international procurement functions, and have documented some of the results of some of those meetings.
    Recommendation: To improve the efficiency and accountability of the emergency food aid procurement process, the Secretary of Agriculture and Administrator of USAID should direct their staffs to work together to take steps to assess WBSCM's functionality by testing the international procurement functions that have been modified since April 2011 and documenting the results.

    Agency: United States Agency for International Development
    Status: Open

    Comments: As of May 2017, USAID has participated in multiple meetings with USDA to assess Web Based Supply Chain Management's (WBSCM) functionality and test the international procurement functions, and are gathering documentation from this process. According to USAID officials, they plan to submit documentation to GAO to close this recommendation by the end of fiscal year 2017.
    Director: Mackin, Michele
    Phone: (202) 512-4841

    2 open recommendations
    including 2 priority recommendations
    Recommendation: To ensure Ford-class carrier acquisitions are supported by sound requirements and a comprehensive testing strategy, and to promote the introduction of reliable, warfighting capable ships into the fleet, the Secretary of Defense should direct the Secretary of the Navy to take the following action prior to accepting delivery of Gerald R. Ford CVN 78. The Secretary of Defense should direct the Secretary of the Navy to conduct a cost-benefit analysis on (1) currently required capabilities, including increased sortie generation rates and reduced manning and (2) the time and money needed to field systems to provide these capabilities, in light of known and projected reliability shortfalls for critical systems. This analysis should be informed by demonstrated system performance from land-based testing, including updated reliability growth projections, and should identify trade space among competing cost, schedule, and performance parameters. The analysis should also consider whether the Navy should seek requirements relief from the Joint Requirements Oversight Council, to the extent necessary, to maximize its return on investment to the warfighter. The Navy should report the results of this analysis to Congress within 30 days of CVN 78 commissioning.

    Agency: Department of Defense
    Status: Open
    Priority recommendation

    Comments: We recommended that DOD conduct a cost-benefit analysis on currently required capabilities, and report the results to Congress within 30 days of CVN 78 commissioning. DOD agreed with our recommendation for a cost-benefit analysis, but disagreed with the timing of it, stating that it plans to measure CVN 78 capabilities through completion of operational testing after ship delivery. Since the release of our report, the Navy completed cost-benefit analyses to determine the acquisition strategy for CVN 79, making two major changes to the ship (replacing the Dual Band Radar (DBR) with a different radar solution and introducing a phased construction and delivery approach.) While these are major program changes, the department did not evaluate the fundamental reason for conducting a cost-benefit analysis, namely that known and projected reliability shortfalls make it unlikely that the program will achieve its sortie generation requirements. In December 2016, an Independent Review Team commissioned by the Under Secretary of Defense, Acquisition, Technology and Logistics completed a comprehensive assessment of the CVN 78's systems, but did not recommend any capability trade-offs or requirements relief.
    Recommendation: To ensure Ford-class carrier acquisitions are supported by sound requirements and a comprehensive testing strategy, and to promote the introduction of reliable, warfighting capable ships into the fleet, the Secretary of Defense should direct the Secretary of the Navy to take the following action prior to accepting delivery of CVN 78. The Secretary of Defense should direct the Secretary of the Navy to adjust the planned post-delivery test schedule to ensure that system integration testing is completed prior to entering initial operational test and evaluation.

    Agency: Department of Defense
    Status: Open
    Priority recommendation

    Comments: Until the Navy updates the test plan in February 2018, we will not know if it will fully address our recommendation. However, recent test schedules suggest an overlap remains between integration testing and the start of initial operational test and evaluation.
    Director: Mackin, Michele
    Phone: (202)512-3000

    1 open recommendations
    Recommendation: The Secretary of Defense should direct the Secretary of the Navy to report to Congress in its annual long-range shipbuilding plan on its plans for a future, larger surface combatant, carrying a more capable version of AMDR and the costs and quantities of this ship.

    Agency: Department of Defense
    Status: Open

    Comments: As in past years, the Congressional Budget Office completed an analysis of the Navy's Fiscal Year 2017 Shipbuilding Plan. The most recent analysis states that the Navy's Long Range Shipbuilding Plan for Fiscal Year 2017 indicates plans to procure 26 DDG 51 Flight III destroyers but provides little specific information regarding a future large surface combatant ship other than noting plans to begin buying the first of 40 ships in 2030 and an estimated cost of $1.9 billion per ship. This lack of detailed information is consistent with past plans. We will keep this recommendation open and will review the Navy's Fiscal Year 2018 plan to determine if greater detail on a future large surface combatant has been developed as a result of changes in the Navy's shipbuilding outlook currently under discussion.
    Director: Gambler, Rebecca S
    Phone: (202)512-3000

    2 open recommendations
    Recommendation: To increase the likelihood of successful implementation of the Arizona Border Surveillance Technology Plan and maximize the effectiveness of technology already deployed, the Commissioner of CBP should take the following step in planning the agency's new technology approach: determine the mission benefits to be derived from implementation of the plan and develop and apply key attributes for metrics to assess program implementation.

    Agency: Department of Homeland Security: United States Customs and Border Protection
    Status: Open

    Comments: As GAO recommended in November 2011, CBP has identified mission benefits to be derived from implementing the Arizona Border Surveillance Technology Plan (Plan). In April 2013, CBP issued its Multi-Year Investment and Management Plan for Border Security Fencing, Infrastructure, and Technology for Fiscal Years 2014 through 2017, which identifies mission benefits to be achieved by all surveillance technologies (e.g., cameras or sensors) to be deployed under the Plan. According to CBP, the majority of these technologies will provide the mission benefits of improved situational awareness and agent safety. Furthermore, according to CBP's Multi-Year Investment and Management Plan for Border Security Fencing, Infrastructure, and Technology for Fiscal Years 2014 - 2017, the technologies deployed or planned for deployment as part of the Plan are intended to help enhance the ability of Border Patrol agents to detect, identify, deter, and respond to threats along the border. CBP's identification of mission benefits will help position CBP to assess its progress in implementing the Plan and the effectiveness of the Plan's technologies in achieving their intended goals. CBP has made some progress in identifying key attributes for metrics to assess implementation of the Arizona Border Surveillance Technology Plan (Plan), as GAO recommended in November 2011, but it has not yet fully identified and applied attributes for metrics for all technologies under the Plan. Since August 2010, CBP has operated multiple technology systems under the Secure Border Initiative Network (SBInet), which preceded the Plan and is a combination of surveillance technologies aimed at creating a "virtual fence" along the southwest border. Specifically, CBP has operated two surveillance systems under SBInet's initial deployment in high-priority regions of the Arizona border. In October 2012, CBP officials stated that these operations identified examples of key attributes for metrics that can be useful in assessing the implementation for technologies. For example, according to CBP, to help measure whether illegal activity has decreased, examples of key attributes include decreases in arrests, complaints by citizens and ranchers, and destruction of public and private lands and property. In November 2014, CBP identified a set of potential key attributes for performance metrics for all technologies to be deployed under the Plan. While CBP has yet to apply these measures, it established a timeline for developing performance measures for each technology. CBP initially expected baselines for each performance measure to be developed by the end of fiscal year 2015. However, in October 2015, CBP officials stated that CBP had modified its time frame for developing baselines and that additional time would be needed to implement and apply key attributes for metrics. In March 2016, CBP officials stated that CBP had planned to use the baseline data to establish a tool by the end of fiscal year 2016. In addition, CBP officials stated these performance measures would profile levels of situational awareness in various areas of the border. In September 2016, CBP provided GAO a case study that assessed technology assist data along with other measures such as field-based assessments of capability gaps, to determine the contributions of surveillance technologies to its mission. While this is a start to developing and applying performance metrics, the case study was limited to one border location and the analysis was limited to select technologies. Until CBP completes its efforts to fully develop and apply key attributes for performance metrics for all technologies to be deployed under the Plan, it will likely not be able to fully assess its progress in implementing the Plan and determine when mission benefits have been fully realized.
    Recommendation: To increase the reliability of CBP's Cost Estimate for the Arizona Border Surveillance Technology Plan, the Commissioner of CBP should update its cost estimate for the Plan using best practices, so that the estimate is comprehensive, accurate, well-documented, and credible. Specifically, the CBP's Office of Technology Innovation and Acquisition (OTIA) program office should (1) fully document data used in the cost model; (2) conduct a sensitivity analysis and risk and uncertainty analysis to determine a level of confidence in the estimate so that contingency funding can be established relative to quantified risk; and (3) independently verify the new life-cycle cost estimate with an independent cost estimate and reconcile any differences.

    Agency: Department of Homeland Security: United States Customs and Border Protection
    Status: Open

    Comments: As GAO recommended in 2011, CBP provided cost estimates for the IFT and RVSS programs, the two highest cost programs in the Arizona Border Surveillance Technology Plan (Plan), in February and March 2012, respectively, and updated the cost estimate for the Plan in June 2013. However, these estimates do not fully meet cost-estimating best practices. In November 2011, GAO reported that the Plan's original cost estimate met some, but not all, cost-estimating best practices. Specifically, CBP had not conducted a sensitivity analysis and a risk and uncertainty analysis to determine a level of confidence in the original estimate, nor did CBP compare the original cost estimate with an independent estimate. For the cost estimate that CBP provided for the IFT in February 2012 and RVSS in March 2012, CBP partially documented the data used in the cost model for the IFT's LCCE (but needs to provide additional data and document management approval) and fully documented the cost model for the RVSS' LCCE. Developing a well-documented cost estimate is a best practice. CBP also conducted a sensitivity analysis and risk and uncertainty analysis to determine the level of confidence in both LCCEs so that contingency funding could be established relative to quantified risk. In addition, CBP's June 2013, CBP revised the cost estimate for the Plan does not fully address our concerns. For example, the IFT and RVSS compose over 90 percent of the Plan's cost in the June 2013 cost estimate; however, CBP has not independently verified its cost estimates for these two programs with independent cost estimates and reconciled any differences. Such action would help CBP better ensure the reliability of each system's cost estimate. Furthermore, the remainder of the June 2013 cost estimate is not fully documented for the Plan's other five programs, consistent with best practices. For example, the estimates for the other five programs are not fully documented because they are provided as summary program costs without detailed descriptions, such as including back-up documentation for labor hours. In November 2015, CBP had yet to update its LCCEs for two of its highest cost programs under the plan. In May 2016, CBP officials stated that the DHS's Cost Analysis Division had started piloting DHS's Independent Cost Estimate capability on the RVSS program in fiscal year 2016. According to CBP officials, this pilot test within CBP is an opportunity to assist DHS in developing its Independent Cost Estimate capability and that CBP selected the RVSS program for the pilot because the RVSS program is at a point in its planning and execution process where it can benefit most from having an independent cost estimate performed as these technologies are being deployed along the southwest border, beyond Arizona. As of October 2016, CBP officials stated that the RVSS schedule and analysis, as well as the results of the independent program cost estimate pilot is expected to be completed at the end of fiscal year 2017. CBP officials stated that they will provide information on the final reconciliation of the independent cost estimate and the RVSS program cost estimate once the pilot has been completed. Further, CBP officials have yet to detail similar plans for the IFT program. As updated life-cycle cost estimates have yet to be completed and independent cost estimates have not been conducted, GAO cannot determine the extent to which the agency is following best practices when updating the life-cycle cost estimates. Moreover, to fully address our recommendation, a LCCE for the Plan that fully addresses best practices is needed to ensure that the estimate is comprehensive, accurate, well-documented, and credible to help the agency and Congress fully understand the impacts of integrating the Plan's various programs.
    Director: Chaplain, Cristina T
    Phone: (202)512-3000

    2 open recommendations
    Recommendation: To strengthen its baselines, facilitate external and independent reviews of those baselines, ensure effective oversight of the BMDS, and further improve transparency and accountability of its efforts, and to improve clarity, consistency, and completeness of the baselines reported to Congress, the Secretary of Defense should ensure that MDA, for resource baselines, obtain independent cost estimates for each baseline.

    Agency: Department of Defense
    Status: Open

    Comments: In providing comments on our report, the agency concurred with this recommendation but has not yet taken all actions necessary to implement it. Although the Missile Defense Agency (MDA) has received independent cost estimates from its internal independent cost group for some programs and components that support the baselines provided in MDA's Ballistic Missile Defense System Accountability Report (BAR), MDA officials told us they have not yet completed independent estimates for all the BAR baselines. In addition, the independent estimates will not have full lifecycle costs which will hamper their effectiveness. We will continue to monitor MDA's progress over the course of our next annual review.
    Recommendation: To strengthen its baselines, facilitate external and independent reviews of those baselines, ensure effective oversight of the BMDS, and further improve transparency and accountability of its efforts, and to improve clarity, consistency, and completeness of the baselines reported to Congress, the Secretary of Defense should ensure that MDA, for schedule baselines, in meeting new statutory requirements to report variances between reported acquisition baselines, also report variances between the test plan as presented in the previous acquisition baseline and the test plan as executed that explain the reason for any changes.

    Agency: Department of Defense
    Status: Open

    Comments: In providing comments on this report, the Department of Defense concurred with our recommendation and has taken initial steps to report the test variances, by laying out the dates of the proposed changes. However, the variances do not include all changes to test objectives, detail when tests are deleted, nor when the altered objectives will be satisfied. MDA has initiated an effort with DOT&E and the OTA to track the movement of test objectives, however these changes are not reported and are only used internally. In addition, MDA utilizes a "mid-year" test change memorandum. The change explains the difference from the prior master test plan, but is not reported. Thus, changes that are included in the mid-year memorandum can not be tracked if one only receives the annual test plan. We will continue to monitor MDA's progress in fiscal year 2017 and determine whether MDA lays out the changes in its upcoming integrated master test plan.
    Director: Mackin, Michele
    Phone: (202) 512-7773

    1 open recommendations
    Recommendation: To capitalize on the increase in knowledge gained by creating new baselines for Deepwater assets, and to better manage acquisitions of further assets and capabilities, the Commandant of the Coast Guard should complete, and present to Congress, a comprehensive review of the Deepwater Program that clarifies the overall cost, schedule, quantities, and mix of assets that are needed to meet mission needs and what trade-offs need to be made considering fiscal constraints, given that the currently approved Deepwater baseline is no longer feasible.

    Agency: Department of Homeland Security: United States Coast Guard
    Status: Open

    Comments: In providing comments on this report, the agency concurred with this recommendation but has not yet taken actions necessary to implement it. Since this report, DHS and the Coast Guard have each completed studies examining the mix of assets that composed the Deepwater Program. To date, the Coast Guard has not yet provided the Congress with a comprehensive review that clarifies the program's cost, schedule, quantities, and mix of assets or takes into account the Coast Guard's needs and available resources and makes recommendations about what trade-offs may be necessary. In 2015, we found that the Coast Guard is currently conducting a fleet-wide analysis, including surface, aviation, and information technology, intended to be a fundamental reassessment of the capabilities and mix of assets the Coast Guard needs to fulfill its missions. The Coast Guard is undertaking this effort consistent with direction from Congress. Specifically, the Coast Guard plans first to rewrite its mission needs statement and concept of operations by 2016. Then, it will use a complex model to develop the full fleet mix study. Based on this, the Coast Guard plans to recommend a set of assets that best meets these needs in terms of capability and cost. The Coast Guard plans to complete the full study in time to inform the fiscal year 2019 budget, though specific dates for these events have not been set forth. As of July 2016, the Coast Guard informed GAO that the modeling is complete and the CONOPS report is being developed with a target date of September 30 for completion.
    Director: Aloise, Eugene E
    Phone: (202)512-6870

    1 open recommendations
    Recommendation: To allow Congress to better oversee management of the nuclear security enterprise and to improve NNSA's management information with respect to the base capabilities necessary to ensure nuclear weapons are safe, secure, and reliable, the Administrator of NNSA should, once the Stockpile Services work breakdown structure reflects a product or capability basis, use this work breakdown structure to develop product/capability cost estimates that adequately justify the congressional budget request for Stockpile Services.

    Agency: Department of Energy: National Nuclear Security Administration
    Status: Open

    Comments: In process: NNSA has significantly improved its work breakdown structure for Stockpile Services. The work breakdown structure now reflects a product or capability basis to a much greater extent than it did previously. NNSA is continuing to work to develop cost estimates for these products and capabilities to adequately justify the congressional budget request for Stockpile Services. In 2014, the Senate Appropriations Committee's Energy and Water Development Subcommittee suggested changes to NNSA's budget structure to align it more closely with products and capabilities beginning with fiscal year 2016 appropriations. GAO will continue to monitor whether NNSA develops cost estimates for Stockpile Services products and capabilities that inform future years' budget requests and justifications.
    Director: Chaplain, Cristina T
    Phone: (202)512-3000

    1 open recommendations
    Recommendation: The Secretary of Defense should direct MDA to ensure that developmental hardware and software changes are not made to the operational baseline that disrupt the assessments needed to understand the capabilities and limitations of new BMDS developments.

    Agency: Department of Defense
    Status: Open

    Comments: In providing comments on this report, the agency concurred with this recommendation. In the June 2010 Ballistic Missile Defense System Accountability Report (BAR), Missile Defense Agency (MDA) provided some operational baselines and continues to do so annually. Nonetheless, configuration changes continue to pose challenges to a thorough assessment of the BMDS architecture. For example, the Director, Operational Test and Evaluation stated that the many configurations of the fielded ground-based interceptor inhibits a full evaluation of the GMD program. Moreover, some changes to BMDS elements are still delivered while testing of the architecture is already underway. We will continue to assess whether MDA fully adopts an approach allowing time for the warfighter and testers to fully understand hardware and software before placing it in the operational baseline.
    Director: Goldenkoff, Robert N
    Phone: (202)512-2757

    1 open recommendations
    including 1 priority recommendation
    Recommendation: To improve the Bureau's use of its master schedule to manage the 2020 decennial census, the Secretary of Commerce should require the Director of the U.S. Census Bureau to include estimates of the resources, such as labor, materials, and overhead costs, in the 2020 integrated schedule for each activity as the schedule is built, and prepare to carry out other steps as necessary to conduct systematic schedule risk analyses on the 2020 schedule.

    Agency: Department of Commerce
    Status: Open
    Priority recommendation

    Comments: Commerce neither agreed nor disagreed with this recommendation. The Bureau continues to refine its 2020 Census master schedule, which it recently announced it completed in July 2016. Bureau officials have periodically described their intent to link resources to activities within their schedules, but as of July 2016 had confirmed that it had not yet done so. The Bureau has provided us with copies of its schedule, but not yet satisfactory evidence of having completed such an analysis. We are beginning an audit of the Bureau's scheduling practices this summer and will review actions the Bureau may have taken to address this recommendation. As of July 2017, we have received initial documents as we begin this review.