Reports & Testimonies

  • GAO’s recommendations database contains report recommendations that still need to be addressed.

    GAO’s recommendations help congressional and agency leaders prepare for appropriations and oversight activities, as well as help improve government operations. Recommendations remain open until they are designated as Closed-implemented or Closed-not implemented. You can explore open recommendations by searching or browsing.

    GAO's priority recommendations are those that we believe warrant priority attention. We sent letters to the heads of key departments and agencies, urging them to continue focusing on these issues. These recommendations are labeled as such. You can find priority recommendations by searching or browsing our open recommendations below, or through our mobile app.

  • Browse Open Recommendations

    Explore priority recommendations by subject terms or browse by federal agency

    Search Open Recommendations

    Search for a specific priority recommendation by word or phrase



  • Governing on the go?

    Our Priorities for Policy Makers app makes it easier for leaders to search our recommendations on the go.

    See the November 10th Press Release


  • Have a Question about a Recommendation?

    • For questions about a specific recommendation, contact the person or office listed with the recommendation.
    • For general information about recommendations, contact GAO's Audit Policy and Quality Assurance office at (202) 512-6100 or apqa@gao.gov.
  • « Back to Results List Sort by   

    Results:

    Subject Term: "Systems development life cycle"

    7 publications with a total of 35 open recommendations including 2 priority recommendations
    Director: Carol Harris
    Phone: (202) 512-4456

    14 open recommendations
    Recommendation: The TSA Administrator should ensure that the TIM program management office establishes and implements specific time frames for determining key strategic implementation details, including how the program will transition from the current state to the final TIM state. (Recommendation 1)

    Agency: Department of Homeland Security
    Status: Open

    Comments: When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.
    Recommendation: The TSA Administrator should ensure that the TIM program management office establishes a schedule that provides planned completion dates based on realistic estimates of how long it will take to deliver capabilities. (Recommendation 2)

    Agency: Department of Homeland Security
    Status: Open

    Comments: When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.
    Recommendation: The TSA Administrator should ensure that the TIM program management office establishes new time frames for implementing the actions identified in the organizational change management strategy and effectively executes against these time frames. (Recommendation 3)

    Agency: Department of Homeland Security
    Status: Open

    Comments: When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.
    Recommendation: The TSA Administrator should ensure that the TIM program management office defines and documents the roles and responsibilities among product owners, the solution team, and any other relevant stakeholders for prioritizing and approving Agile software development work. (Recommendation 4)

    Agency: Department of Homeland Security
    Status: Open

    Comments: When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.
    Recommendation: The TSA Administrator should ensure that the TIM program management office establishes specific prioritization levels for current and future features and user stories. (Recommendation 5)

    Agency: Department of Homeland Security
    Status: Open

    Comments: When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.
    Recommendation: The TSA Administrator should ensure that the TIM program management office implements automated Agile management testing and deployment tools, as soon as possible. (Recommendation 6)

    Agency: Department of Homeland Security
    Status: Open

    Comments: When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.
    Recommendation: The TSA Administrator should ensure that the TIM program management office updates the Systems Engineering Life Cycle Tailoring Plan to reflect the current governance framework and milestone review processes. (Recommendation 7)

    Agency: Department of Homeland Security
    Status: Open

    Comments: When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.
    Recommendation: The TSA Administrator should ensure that the TIM program management office establishes thresholds or targets for acceptable performance-levels. (Recommendation 8)

    Agency: Department of Homeland Security
    Status: Open

    Comments: When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.
    Recommendation: The TSA Administrator should ensure that the TIM program management office begins collecting and reporting on Agile-related cost metrics. (Recommendation 9)

    Agency: Department of Homeland Security
    Status: Open

    Comments: When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.
    Recommendation: The TSA Administrator should ensure that the TIM program management office ensures that program velocity is measured and reported consistently. (Recommendation 10)

    Agency: Department of Homeland Security
    Status: Open

    Comments: When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.
    Recommendation: The TSA Administrator should ensure that the TIM program management office ensures that unit test coverage for software releases is measured and reported accurately. (Recommendation 11)

    Agency: Department of Homeland Security
    Status: Open

    Comments: When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.
    Recommendation: The Secretary of Homeland Security should direct the Under Secretary for Management to ensure that appropriate DHS leadership reaches consensus on needed oversight and governance changes related to the frequency of reviewing Agile programs, and then documents and implements associated changes. (Recommendation 12)

    Agency: Department of Homeland Security
    Status: Open

    Comments: When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.
    Recommendation: The Secretary of Homeland Security should direct the Under Secretary for Management to ensure that the Office of the Chief Technology Officer completes guidance for Agile programs to use for collecting and reporting on performance metrics. (Recommendation 13)

    Agency: Department of Homeland Security
    Status: Open

    Comments: When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.
    Recommendation: The Secretary of Homeland Security should direct the Under Secretary for Management to ensure that DHS-level oversight bodies review key Agile performance and cost metrics for the TIM program and use them to inform management oversight decisions. (Recommendation 14)

    Agency: Department of Homeland Security
    Status: Open

    Comments: When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.
    Director: Shelby S. Oakley
    Phone: (202) 512-3841

    1 open recommendations
    Recommendation: The NNSA Deputy Administrator for DNN should revise the DNN program management policy to require DNN programs and subprograms to follow life-cycle program management. These requirements should include development of schedule and cost estimates that cover the life cycle of DNN programs and subprograms, use of methods to account for uncertainty and risk in such estimates, use of cost and schedule baselines to measure performance over program and subprogram life cycles, and development of program management plans. (Recommendation 1)

    Agency: Department of Energy: National Nuclear Security Administration: Office of Defense Nuclear Nonproliferation
    Status: Open

    Comments: When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.
    Director: Valerie C. Melvin
    Phone: (202) 512-6304

    8 open recommendations
    Recommendation: To assist VA in sustaining an IT workforce with the necessary knowledge, skills, and abilities to execute its mission and goals, the Secretary of Veterans Affairs should direct the Chief Information Officer to track and review OI&T historical workforce data and projections related to leadership retirements.

    Agency: Department of Veterans Affairs
    Status: Open

    Comments: VA concurred with our recommendation and reported that OI&T's Human Capital Management Office (HCM) had completed a succession planning project that encompassed all senior leadership and included data review and risk assessment for each position. VA also stated that OI&T tracks the gains and losses associated with its leadership positions and provided this information for fiscal year 2016. However, the department has not provided documentation that supports the assertion that historical and projected OI&T leadership retirement data was presented and discussed as part of the succession planning project and did not provide data on projected retirements for OI&T's leadership positions. Additionally, the department stated that OI&T HCM has the ability to project retirement eligibility but has not provided documentation to support this assertion. It is important that VA tracks and reviews its OI&T historical workforce data and forecasts its leadership retirements to avoid being unprepared to effectively respond to vacancies in key leadership positions.
    Recommendation: To assist VA in sustaining an IT workforce with the necessary knowledge, skills, and abilities to execute its mission and goals, the Secretary of Veterans Affairs should direct the Chief Information Officer to identify IT skills needed beyond the current fiscal year to assist in identifying future skills gaps.

    Agency: Department of Veterans Affairs
    Status: Open

    Comments: VA concurred with our recommendation and reported that Information Technology Workforce Development (ITWD) will produce reports that identify skill gaps and will contain long-term recommendations that show the types of IT skills each organization needs to increase and which proficiency level targets need the most emphasis. As of July 2017, VA stated that ITWD reviewed, and updated where needed, the fiscal year 2017 competencies within each OI&T competency model role in order to align the models to the OI&T Transformation initiative. According to the department, the resulting updates support learning solutions that sustain and accelerate OI&T's transformation. Additionally, VA stated that 85 percent of OI&T staff completed a validated competency self-assessment and provided the OI&T fiscal year 2017 Training Gap Analysis Report which shows the strengths and gaps of OI&T by organization, trends between fiscal years 2016 and 2017, findings, next steps, and recommended actions for the next fiscal year. The department also stated that ITWD held meetings to review skill gap and learning solution reports. VA provided these reports and they present the top gaps and strengths, key findings, and next steps to address the skill gaps. While the department has taken these actions, its OI&T Training Gap Analysis Report does not identify IT skills needed beyond fiscal year 2017.
    Recommendation: To assist VA in establishing comprehensive and documented processes that reflect system development and acquisition best practices, the Secretary of Veterans Affairs should direct the Chief Information Officer to revise OI&T's documented processes related to project planning, to include (1) estimating the level of effort that will need to be expended for work products and tasks, and (2) making adjustments to the project plan to reconcile differences between estimated and available resources.

    Agency: Department of Veterans Affairs
    Status: Open

    Comments: VA concurred with our recommendation and stated that OI&T is documenting changes to processes related to project planning as it transitions from PMAS to the Veteran-Focused Integration Process (VIP). According to VA, the VIP processes will lead to better requirements elaboration and prioritization, increasing significantly the accuracy of estimates related to level of effort. Additionally, the department stated that by using short Agile sprints, the project team will be able adjust the project plan frequently to reconcile differences between estimated and available resources. As of July 2017, VA stated that all projects have transitioned to the VIP, which ensures they are incorporating the Agile methodology into the project lifecycle. According to the department, the latest version of its VIP Guide incorporates the use of daily scrum and weekly scrum of scrum meetings that can be used to frequently adjust the project plan to reconcile differences between estimated and available resources. VA stated that the project planning processes will continue to evolve beyond July and expects to complete its actions in response to this recommendation by the end of fiscal year 2017.
    Recommendation: To assist VA in establishing comprehensive and documented processes that reflect system development and acquisition best practices, the Secretary of Veterans Affairs should direct the Chief Information Officer to revise OI&T's documented processes related to requirements management, to include identifying changes to be made to plans and work products as a result of requirements baseline changes.

    Agency: Department of Veterans Affairs
    Status: Open

    Comments: VA concurred with our recommendation and reported that OI&T is revising its documentation related to requirements management as part of the transition to the Veteran-Focused Integration Process (VIP). According to VA, requirements will be tracked using the IBM Rational Tools Suite, which will be able to provide a snapshot of the original baseline and all captured changes in the form of an audit trail that captures the history of requirement changes. As of July 2017, the department stated that all projects have transitioned to the VIP and requirements baselines and subsequent changes are tracked in the Rational Tools Suite. VA also reported that efforts in fiscal year 2017 to consolidate all mandatory architectural, design, and process methodologies into a single library of requirements were successful, which resulted in combining the full body of requirements. Additionally according to the department, versioning of the requirements will allow the office to trace specific versions of individual requirements and their evolution by time period and project inheritance. VA stated that it expects to complete its actions in response to this recommendation by the end of fiscal year 2017.
    Recommendation: To assist VA in establishing comprehensive and documented processes that reflect system development and acquisition best practices, the Secretary of Veterans Affairs should direct the Chief Information Officer to revise OI&T's documented processes related to risk management, to include (1) determining costs and benefits of implementing the risk mitigation plan for each risk and (2) collecting performance measures on risk handling activities.

    Agency: Department of Veterans Affairs
    Status: Open

    Comments: VA concurred with our recommendation and reported that the IBM Rational Tools Suite will be used to manage risks and issues. According to VA, the tools suite will allow requirements to be linked to risks, which will provide traceability; teams will be able to track and report steps taken to mitigate risks; and an audit trail will show the history of changes made to each risk. The department also reported that the Office of Privacy and Risk will establish risk mitigation strategies for OI&T. As of July 2017, VA stated that risks data capture has been developed as a standardized process and that data on project and program risks in the Rational Tools Suite is aggregated and prepared for use to verify aggressive management, and will be included in enterprise reporting. The department stated that work is underway with the Performance Management Office and that OI&T expects to complete its actions in response to this recommendation by the end of fiscal year 2017.
    Recommendation: To assist VA in establishing comprehensive and documented processes that reflect system development and acquisition best practices, the Secretary of Veterans Affairs should direct the Chief Information Officer to revise OI&T's documented processes related to project monitoring and control, to include the 10 best practices that were missing from the guidance.

    Agency: Department of Veterans Affairs
    Status: Open

    Comments: VA concurred with our recommendation and reported that implementation of the Veteran-Focused Integration Process (VIP) and Agile processes within OI&T will address eight of the ten best practices related to project monitoring and control that were missing from its guidance. In regard to monitoring the knowledge and skills of project staff, OI&T's IT Workforce Development (ITWD) group collects and analyzes competency assessment data, which is used in requirements gathering meetings with OI&T leaders. According to VA, during these meetings organizational needs and next steps are discussed in detail. Additionally, the department's latest version of its VIP Guide states that the product team should be cross-functional and include all skills needed to deliver a product. Further, the department reported that data management activities, issues, and impacts will be managed using VIP, Agile, and IBM Rational Tools Suite. According to its VIP Guide, OI&T expects that all products follow the Agile product management process and use the Rational Tools Suite to manage scheduled product sprints and backlog, product requirements, risks and issues, and product planning and engineering documentation, among others. Also, VA stated that Agile methodologies will require stakeholders to be involved in the daily scrum meetings, user acceptance testing, and acceptance of deliverables, which will address stakeholders being involved regularly and documenting the results of stakeholder involvement status reviews. According to the VIP Guide, the Agile development methodologies require development teams to meet often with stakeholders to ensure transparency and foster a collaborative work environment. Additionally, the department stated that critical decision events are using Rational based data assessments to report on level of satisfaction of project controls and process compliance requirements. Further, according to the VIP Guide, the Product Owner will have a key role in the decision-making process during the development of the product and will be able to regularly express concerns and/or approvals to best meet user satisfaction. The department stated that critical decision events are being held at the portfolio level, and action items from these events are being tracked. VA provided meeting minutes from critical decision events that were held in October and December 2016. The December 2016 meeting minutes identified action items and the status of those items. Although VA has taken actions to address the majority of best practices related to project monitoring and control, the department's new VIP process does not include two practices that call for (1) tracking expended effort and (2) monitoring the utilization of staff and resources. Until OI&T's documented processes for project monitoring and control fully reflect best practices, the office is at risk that its projects will not achieve expected results.
    Recommendation: To assist VA in establishing comprehensive and documented processes that reflect system development and acquisition best practices, the Secretary of Veterans Affairs should direct the Chief Information Officer to revise OI&T's documented processes related to process and product quality assurance, to include (1) documenting a description of the quality assurance reporting chain and defining how objectivity will be ensured, and (2) periodically reviewing open noncompliance issues and trends with management that is designated to receive and act on them.

    Agency: Department of Veterans Affairs
    Status: Open

    Comments: VA concurred with our recommendation and reported that the implementation of the Veteran-Focused Integration Process (VIP), Agile processes, and the Rational Toolset within OI&T will address process and product quality assurance. According to VA, as a part of VIP, the Product Owner is engaged from intake through project completion, which will ensure that the quality of the product is maintained throughout the life cycle. Additionally the department reported that the process of periodically reviewing open non-compliance issues and trends with management that is designated to receive and act on them will be accomplished through CIOStat meetings held with OI&T senior leadership. VA also reported that the Rational Quality Manager tool is used to automate routine testing activities to identify non-compliance issues and trends. As of July 2017, the department stated that the Product Owner is beginning to have a stronger role on the project team, which enables them to assist in all types of issues, including quality assurance. VA also stated that Release Agents develop and distribute Release Readiness Reports, which provide a status of all release requirements and of traceability among requirements, deliverables, and test results. VA expects to complete its actions in response to this recommendation by the end of fiscal year 2017.
    Recommendation: To assist VA in establishing comprehensive and documented processes that reflect system development and acquisition best practices, the Secretary of Veterans Affairs should direct the Chief Information Officer to revise OI&T's documented processes related to project scheduling, to include the 9 best practices that were missing from the guidance and revise the documented processes where the guidance was contrary to best practices.

    Agency: Department of Veterans Affairs
    Status: Open

    Comments: VA concurred with our recommendation and reported that the implementation of VIP and Agile processes within OI&T will address five of the nine best practices related to project scheduling that are missing from its guidance. According to VA, business and compliance requirements will be captured during the planning phase and maintained in the IBM Rational Tools Suite to manage scheduled project/product builds and backlog which will allow the project to more accurately maintain the schedule baseline, capture all schedule changes, and provides an audit trail of all the changes. Additionally, the department reported that the IBM Rational Tools Suite connects requirements, change orders, test cases, and test results in order to have full traceability in a closed loop system. VA also noted that the use of short development builds within Agile increases the probability of successful adherence to the schedule; and Agile provides the flexibility to make schedule changes using the backlog to prioritize requirements. As of July 2017, VA stated that Project Build Planning sessions capture and prioritize all backlog items with high level activities captured in the VIP Dashboard; and that each project task receives an estimated duration. The department also stated that the project team commits to a high level scope for each build and then the scope is solidified and committed to in detail at each Sprint Plan. According to VA, at the end of each sprint the Product Owner accepts or rejects the product of what was committed to at Sprint Planning. The department also stated that there is a high-level commitment at the Critical Decision 1 meeting; that each build gets committed to at a more granular level; and that sprint planning includes establishing a firm commitment for exactly what will be completed during the sprint. The department further stated that part of the Agile process being used by OI&T removes rigid, mandatory constraints as long as project teams follow compliance epics. Additionally, the department reported that because of the use of Agile methodology, if a task is critical today, the project team can reprioritize and address the needs of the project immediately. According to VA, Agile supports both sustainment and development projects, by allowing changes to the project backlog to address high priority functionality. VA also stated that Agile allows flexibility to shift from one build to another based on priorities and to shift backlog items based on VIP Triad priorities. Additionally, according to the department, risks are managed in the Rational Tools Suite and impediments are raised and escalated during daily scrums and scrum of scrum calls. The VIP Guide indicates that product teams are required to make timely updates to the VIP Dashboard regarding schedule and that the Rational Tools Suite will be used to manage and administer source control and baselines; manage risks and issues; and manage scheduled product sprints and backlogs. However, the VIP Guide does not include practices to (1) document that each project task should receive a duration estimate; (2)require that the project schedule be traceable horizontally and vertically; (3) sequence all activities; and (4) confirm that the critical path is valid. Until OI&T's documented processes for developing schedules fully reflect best practices, the office is at risk that schedules created for its projects will not be reliable.
    Director: David C. Trimble
    Phone: (202) 512-3841

    2 open recommendations
    Recommendation: To develop reliable cost estimates for the TRU waste removal project and for the TWF construction project at LANL, the Secretary of Energy should direct NNSA and the Office of Environmental Management to revise the cost estimate for the TRU waste removal project to ensure that it uses updated assumptions based on the current understanding of project conditions, such as the status of WIPP.

    Agency: Department of Energy
    Status: Open

    Comments: As of April 2017, Department of Energy (DOE) officials indicated that a revised life-cycle baseline cost estimate was prepared for all Office of Environmental Management mission work at Los Alamos National Laboratory, including transuranic waste removal work. DOE approved the revised cost estimate in July 2016. After we review documentation of the estimate, we will evaluate whether it is sufficient to close the recommendation.
    Recommendation: To develop reliable cost estimates for the TRU waste removal project and for the TWF construction project at LANL, the Secretary of Energy should direct NNSA to revise and update the TWF project's cost estimate by following all best practices for developing a reliable cost estimate that covers all life-cycle costs for better managing the project going forward.

    Agency: Department of Energy
    Status: Open

    Comments: The Department of Energy (DOE) agreed with the recommendation. As of March 2017, DOE indicated that Los Alamos National Laboratory prepared a cost estimate for the operations and maintenance of the Transuranic Waste Facility (TWF) facility in December 2015, which was reviewed and accepted by the responsible program offices. DOE indicated that the revised estimate reflected operational costs for a seven-year window and incorporated applicable best practices, including documentation of any significant deviations and uncertainties impacting the estimate, among other things. After we obtain documentation of the estimate, we will evaluate the action to determine whether it is sufficient to close the recommendation as implemented.
    Director: Cary Russell
    Phone: (202) 512-5431

    6 open recommendations
    including 1 priority recommendation
    Recommendation: To help DOD develop an affordable sustainment strategy for the F-35, the Secretary of Defense should direct the Under Secretary of Defense for Acquisitions, Technology and Logistics to direct the F-35 Program Executive Officer to establish affordability constraints linked to, and informed by, military service budgets that will help guide sustainment decisions, prioritize requirements, and identify additional areas for savings by March 2015, at which point the Future Support Construct decision will be approved.

    Agency: Department of Defense
    Status: Open
    Priority recommendation

    Comments: DOD concurred with our recommendation and stated in April 2017 that the F-35 Program Executive Officer and the F-35 enterprise have expanded their collaborative effort to reduce F-35 operating and support (O&S) costs to ensure that they deliver affordable readiness for the F-35 fleet. In an effort to reduce overall O&S costs, the department has undertaken several initiatives. For example, according to DOD, as of January 2017, a program office "cost war room" initiative had reduced the 2012 F-35 annual cost estimate by $60.7 billion. Additionally, according to DOD, a Reliability and Maintainability Improvement Program has resulted in a $1.7 billion O&S cost avoidance through the program's life cycle. Other efforts are also under way that aim to help reduce O&S costs by better informing sustainment decision-making. While the department is taking steps to try to reduce overall O&S costs, the program has yet to develop affordability constraints linked to the military services' budgets. Without affordability constraints that are linked to military service budgets, it remains unclear the extent to which the military services can afford to operate and sustain the F-35 throughout its life cycle as currently planned.
    Recommendation: To help DOD address key risks to F-35 affordability and operational readiness, and to improve the reliability of its O&S cost estimates for the life cycle of the program, the Secretary of Defense should direct the F-35 Program Executive Officer, to enable DOD to better identify, address, and mitigate performance issues with the Autonomic Logistics Information System (ALIS) that could have an effect on affordability, as well as readiness, to establish a performance-measurement process for ALIS that includes, but is not limited to, performance metrics and targets that (1) are based on intended behavior of the system in actual operations and (2) tie system performance to user requirements.

    Agency: Department of Defense
    Status: Open

    Comments: According to DOD officials, the ALIS Integrated Product Team (IPT) is continuing to work with the Joint Program Office's Performance Based Logistics (PBL) team to further develop and refine appropriate metrics for inclusion into future sustainment contracts. Although DOD has made progress in developing performance metrics for ALIS, as of September 2017, DOD has yet to develop metrics that are based on intended behavior of the system and tie system performance to user requirements. Until this progression is made, this recommendation will remain open.
    Recommendation: To help DOD address key risks to F-35 affordability and operational readiness, and to improve the reliability of its O&S cost estimates for the life cycle of the program, the Secretary of Defense should direct the F-35 Program Executive Officer, to develop a high level of confidence that the aircraft will achieve its R+M goals, to develop a software reliability and maintainability (R+M) assessment process, with metrics, by which the program can monitor and determine the effect that software issues may have on overall F-35 R+M issues.

    Agency: Department of Defense
    Status: Open

    Comments: DOD has an R&M assessment process in place, but as of September 2017, had not developed a process that would focus directly on software reliability and maintainability. Until DOD develops a process more focused on software and its effects on overall R&M issues, this recommendation will remain open.
    Recommendation: To help DOD address key risks to F-35 affordability and operational readiness, and to improve the reliability of its O&S cost estimates for the life cycle of the program, the Secretary of Defense should direct the F-35 Program Executive Officer, to promote competition, address affordability, and inform its overarching sustainment strategy, to develop a long-term Intellectual Property (IP) Strategy to include, but not be limited to, the identification of (1) current levels of technical data rights ownership by the federal government and (2) all critical technical data needs and their associated costs.

    Agency: Department of Defense
    Status: Open

    Comments: DOD has still not developed an overall strategy that would identify data rights ownership, needs, and costs. As of September 2017, the program had taken some steps to develop an Intellectual Property Strategy, but has not identified all critical needs and their associated costs. Program office officials said that they are currently working with the prime contractor to develop a list of technical data requirements. Until this strategy is developed, this recommendation will remain open.
    Recommendation: To help DOD address key risks to F-35 affordability and operational readiness, and to improve the reliability of its O&S cost estimates for the life cycle of the program, the Secretary of Defense should direct the F-35 Program Executive Officer, to understand the potential range of costs associated with the JPO F-35 O&S cost estimate, to conduct uncertainty analyses on future JPO estimates.

    Agency: Department of Defense
    Status: Open

    Comments: As of September 2017, DOD had not applied risk/uncertainty analyses to its cost estimates. Until it does so, this recommendation will remain open.
    Recommendation: To improve the reliability of the CAPE F-35 O&S cost estimate, the Secretary of Defense should direct the Director of CAPE, for future F-35 O&S cost estimates, to conduct uncertainty analyses to understand the potential range of costs associated with its estimates to reflect the most likely costs associated with the program.

    Agency: Department of Defense
    Status: Open

    Comments: According to DOD officials, the Cost Assessment and Program Evaluation (CAPE) has not updated its F-35 estimate subsequent to the release of GAO-14-778. Pending a major program change, CAPE will update the F-35 O&S estimate for the full-rate production decision point in the second quarter of fiscal year 2019. Until CAPE updates its F-35 estimate, we will not be able to determine if they will perform any uncertainty analyses on its cost estimate; therefore, this recommendation will remain open as of September 1, 2017.
    Director: Valerie C. Melvin
    Phone: (202) 512-6304

    2 open recommendations
    including 1 priority recommendation
    Recommendation: To ensure effective management and modernization of HUD's IT environment, the Secretary of Housing and Urban Development should direct the department's Chief Information Officer to establish a means for evaluating progress toward institutionalizing management controls and commit to time lines for activities and next steps.

    Agency: Department of Housing and Urban Development
    Status: Open

    Comments: As of April 2017, HUD had not yet established a means for evaluating progress toward institutionalizing IT management controls. According to HUD officials, the department expects to evaluate the controls through an update to its IT Management Framework scheduled to be completed during fiscal year 2017.
    Recommendation: To ensure effective management and modernization of HUD's IT environment, the Secretary of Housing and Urban Development should direct the department's Chief Information Officer to define the scope, implementation strategy, and schedule of its overall modernization approach, with related goals and measures for effectively overseeing the effort.

    Agency: Department of Housing and Urban Development
    Status: Open
    Priority recommendation

    Comments: In August 2016, HUD officials reported that the department was taking actions intended to establish a new, stronger enterprise approach for IT development and operations. As of April 2017, the department reported that it was in phase 2 of a 4-phase application assessment initiative expected to address this recommendation. However, HUD has not yet provided evidence of how the new approach is expected to define the scope, implementation strategy, and schedule for modernizing the department's IT.
    Director: Gambler, Rebecca S
    Phone: (202)512-3000

    2 open recommendations
    Recommendation: To increase the likelihood of successful implementation of the Arizona Border Surveillance Technology Plan and maximize the effectiveness of technology already deployed, the Commissioner of CBP should take the following step in planning the agency's new technology approach: determine the mission benefits to be derived from implementation of the plan and develop and apply key attributes for metrics to assess program implementation.

    Agency: Department of Homeland Security: United States Customs and Border Protection
    Status: Open

    Comments: As GAO recommended in November 2011, CBP has identified mission benefits to be derived from implementing the Arizona Border Surveillance Technology Plan (Plan). In April 2013, CBP issued its Multi-Year Investment and Management Plan for Border Security Fencing, Infrastructure, and Technology for Fiscal Years 2014 through 2017, which identifies mission benefits to be achieved by all surveillance technologies (e.g., cameras or sensors) to be deployed under the Plan. According to CBP, the majority of these technologies will provide the mission benefits of improved situational awareness and agent safety. Furthermore, according to CBP's Multi-Year Investment and Management Plan for Border Security Fencing, Infrastructure, and Technology for Fiscal Years 2014 - 2017, the technologies deployed or planned for deployment as part of the Plan are intended to help enhance the ability of Border Patrol agents to detect, identify, deter, and respond to threats along the border. CBP's identification of mission benefits will help position CBP to assess its progress in implementing the Plan and the effectiveness of the Plan's technologies in achieving their intended goals. CBP has made some progress in identifying key attributes for metrics to assess implementation of the Arizona Border Surveillance Technology Plan (Plan), as GAO recommended in November 2011, but it has not yet fully identified and applied attributes for metrics for all technologies under the Plan. Since August 2010, CBP has operated multiple technology systems under the Secure Border Initiative Network (SBInet), which preceded the Plan and is a combination of surveillance technologies aimed at creating a "virtual fence" along the southwest border. Specifically, CBP has operated two surveillance systems under SBInet's initial deployment in high-priority regions of the Arizona border. In October 2012, CBP officials stated that these operations identified examples of key attributes for metrics that can be useful in assessing the implementation for technologies. For example, according to CBP, to help measure whether illegal activity has decreased, examples of key attributes include decreases in arrests, complaints by citizens and ranchers, and destruction of public and private lands and property. In November 2014, CBP identified a set of potential key attributes for performance metrics for all technologies to be deployed under the Plan. While CBP has yet to apply these measures, it established a timeline for developing performance measures for each technology. CBP initially expected baselines for each performance measure to be developed by the end of fiscal year 2015. However, in October 2015, CBP officials stated that CBP had modified its time frame for developing baselines and that additional time would be needed to implement and apply key attributes for metrics. In March 2016, CBP officials stated that CBP had planned to use the baseline data to establish a tool by the end of fiscal year 2016. In addition, CBP officials stated these performance measures would profile levels of situational awareness in various areas of the border. In September 2016, CBP provided GAO a case study that assessed technology assist data along with other measures such as field-based assessments of capability gaps, to determine the contributions of surveillance technologies to its mission. While this is a start to developing and applying performance metrics, the case study was limited to one border location and the analysis was limited to select technologies. Until CBP completes its efforts to fully develop and apply key attributes for performance metrics for all technologies to be deployed under the Plan, it will likely not be able to fully assess its progress in implementing the Plan and determine when mission benefits have been fully realized.
    Recommendation: To increase the reliability of CBP's Cost Estimate for the Arizona Border Surveillance Technology Plan, the Commissioner of CBP should update its cost estimate for the Plan using best practices, so that the estimate is comprehensive, accurate, well-documented, and credible. Specifically, the CBP's Office of Technology Innovation and Acquisition (OTIA) program office should (1) fully document data used in the cost model; (2) conduct a sensitivity analysis and risk and uncertainty analysis to determine a level of confidence in the estimate so that contingency funding can be established relative to quantified risk; and (3) independently verify the new life-cycle cost estimate with an independent cost estimate and reconcile any differences.

    Agency: Department of Homeland Security: United States Customs and Border Protection
    Status: Open

    Comments: As GAO recommended in 2011, CBP provided cost estimates for the IFT and RVSS programs, the two highest cost programs in the Arizona Border Surveillance Technology Plan (Plan), in February and March 2012, respectively, and updated the cost estimate for the Plan in June 2013. However, these estimates do not fully meet cost-estimating best practices. In November 2011, GAO reported that the Plan's original cost estimate met some, but not all, cost-estimating best practices. Specifically, CBP had not conducted a sensitivity analysis and a risk and uncertainty analysis to determine a level of confidence in the original estimate, nor did CBP compare the original cost estimate with an independent estimate. For the cost estimate that CBP provided for the IFT in February 2012 and RVSS in March 2012, CBP partially documented the data used in the cost model for the IFT's LCCE (but needs to provide additional data and document management approval) and fully documented the cost model for the RVSS' LCCE. Developing a well-documented cost estimate is a best practice. CBP also conducted a sensitivity analysis and risk and uncertainty analysis to determine the level of confidence in both LCCEs so that contingency funding could be established relative to quantified risk. In addition, CBP's June 2013, CBP revised the cost estimate for the Plan does not fully address our concerns. For example, the IFT and RVSS compose over 90 percent of the Plan's cost in the June 2013 cost estimate; however, CBP has not independently verified its cost estimates for these two programs with independent cost estimates and reconciled any differences. Such action would help CBP better ensure the reliability of each system's cost estimate. Furthermore, the remainder of the June 2013 cost estimate is not fully documented for the Plan's other five programs, consistent with best practices. For example, the estimates for the other five programs are not fully documented because they are provided as summary program costs without detailed descriptions, such as including back-up documentation for labor hours. In November 2015, CBP had yet to update its LCCEs for two of its highest cost programs under the plan. In May 2016, CBP officials stated that the DHS's Cost Analysis Division had started piloting DHS's Independent Cost Estimate capability on the RVSS program in fiscal year 2016. According to CBP officials, this pilot test within CBP is an opportunity to assist DHS in developing its Independent Cost Estimate capability and that CBP selected the RVSS program for the pilot because the RVSS program is at a point in its planning and execution process where it can benefit most from having an independent cost estimate performed as these technologies are being deployed along the southwest border, beyond Arizona. As of October 2016, CBP officials stated that the RVSS schedule and analysis, as well as the results of the independent program cost estimate pilot is expected to be completed at the end of fiscal year 2017. CBP officials stated that they will provide information on the final reconciliation of the independent cost estimate and the RVSS program cost estimate once the pilot has been completed. Further, CBP officials have yet to detail similar plans for the IFT program. As updated life-cycle cost estimates have yet to be completed and independent cost estimates have not been conducted, GAO cannot determine the extent to which the agency is following best practices when updating the life-cycle cost estimates. Moreover, to fully address our recommendation, a LCCE for the Plan that fully addresses best practices is needed to ensure that the estimate is comprehensive, accurate, well-documented, and credible to help the agency and Congress fully understand the impacts of integrating the Plan's various programs.