Quadrennial Homeland Security Review:

Improved Risk Analysis and Stakeholder Consultations Could Enhance Future Reviews

GAO-16-371: Published: Apr 15, 2016. Publicly Released: Apr 15, 2016.

Additional Materials:

Contact:

Andrew Von Ah
(213) 830-1011
vonaha@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

What GAO Found

The Department of Homeland Security (DHS) assessed risk for the second Quadrennial Homeland Security Review (QHSR) and considered threats, vulnerabilities, and consequences; however, DHS did not document how its various analyses were synthesized to generate results, thus limiting the reproducibility and defensibility of the results. Without sufficient documentation, the QHSR risk results cannot easily be validated or the assumptions tested, hindering DHS's ability to improve future assessments. In addition, the QHSR describes homeland security hazards, but does not rank those hazards or provide prioritized strategies to address them. Comparing and prioritizing risks helps identify where risk mitigation is most needed and helps justify cost-effective risk management options. Without determining prioritized risk outcomes, DHS is missing an opportunity to more efficiently mitigate risk or identify the resources required for addressing different levels and types of risks.

The President's fiscal years 2015 and 2016 budget requests for DHS were generally presented in alignment with the QHSR. However, DHS has faced challenges accounting for its spending by mission, which it is taking actions to address, such as developing a new common appropriation structure to better link the department's funding request to the execution of its missions. DHS also developed performance measures for all of the QHSR mission areas.

DHS expanded its stakeholder outreach efforts, but 43 of 61 stakeholders who provided narrative responses to one question in GAO's survey stated that collaboration with stakeholders could be improved. For example, one respondent reported that stakeholders were asked to react to information provided by DHS rather than assist in formulating the QHSR approach and execution. DHS officials reported being limited by staff, time, and other constraints, and thus directed stakeholders to provide feedback via various web-based forums. Although the online forums allowed DHS to reach 2,000 representatives during its 2014 QHSR development process, DHS's QHSR After Action Report noted that the tools were used to validate study findings instead of informing them. Without fostering interactive communication, DHS may miss opportunities to incorporate stakeholder perspectives from the entire homeland security enterprise and thereby may not have fully informed the QHSR effort.

2014 Quadrennial Homeland Security Review Development Timeline

2014 Quadrennial Homeland Security Review Development Timeline

Why GAO Did This Study

Homeland security threats continue to evolve and include challenges ranging from terrorist attacks to natural disasters, emphasizing the need for DHS to periodically examine and strengthen the nation's homeland security strategy. Further, the Implementing Recommendations of the 9/11 Commission Act of 2007 requires DHS to conduct such a review every 4 years. GAO reported in 2011 on DHS's first QHSR and recommended DHS provide sufficient time for stakeholder consultations and examine how risk information could be used to prioritize mission efforts. DHS issued its second QHSR in 2014.

GAO was asked to assess the QHSR. This report addresses the extent to which DHS (1) examined and used risk information to inform the QHSR and its implementation, (2) aligned its budget and performance measures to QHSR mission goals, and (3) collaborated with stakeholders to develop the QHSR. GAO analyzed relevant statutes and QHSR documentation; conducted a nongeneralizable survey of DHS-identified federal and nonfederal QHSR stakeholders, receiving responses from 93 of the 182 QHSR stakeholders it contacted; and interviewed officials from DHS, federal, and nonfederal entities.

What GAO Recommends

GAO is making four recommendations, including that for future QHSRs DHS improve its risk assessment documentation, prioritize risks, and ensure stakeholder meetings are interactive. DHS concurred with our recommendations.

For more information, contact Andrew Von Ah at (213) 830-1011 or vonaha@gao.gov.

Recommendations for Executive Action

  1. Status: Open

    Comments: As of June 2016, the Office of Policy's Office of Strategy, Plans, Analysis and Risks completed initial meetings in April 2016 with government and non-government subject matter experts to refine risk analyses for the upcoming 2018 QHSR. Representatives from the department's component and headquarters staff are to take part in the Department's Risk Modeling and Analysis Steering Committee by reviewing, documenting and approving proposed new methodologies planned to help identify and prioritize threats and hazards. This effort is intended to lead to a documented, reproducible, and defensible assessment, according to the DHS officials. This recommendation will remain open until we verify that the risk analysis contains these elements.

    Recommendation: To ensure the quality of the risk assessments used to inform its future QHSR processes, the Secretary of Homeland Security should direct the Assistant Secretary for Policy to ensure future QHSR risk assessment methodologies reflect key elements of successful risk assessment methodologies, such as being: (1) Documented, which includes documenting how risk information was integrated to arrive at the assessment results, (2) Reproducible, which includes producing comparable, repeatable results, and (3) Defensible, which includes communicating any implications of uncertainty to users of the risk results.

    Agency Affected: Department of Homeland Security

  2. Status: Open

    Comments: As of June 2016, the Office of Policy's Office of Strategy, Plans, Analysis, and Risk, with support from the RAND Corporation, has proposed a methodology to assess threats, hazards, and vulnerabilities impacting U.S. homeland security. In addition, the department's Risk Modeling and Analysis Executive Steering Committee is to review and approve the proposed methodology. The methodology is intended to enable the Department of Homeland Security to compare and prioritize homeland security risks and risk mitigation strategies, according to DHS officials. The recommendation will remain open until we verify that the methodology allows such comparisons.

    Recommendation: To enable the use of risk information in supporting resource allocation decisions, guiding investments, and highlighting the measures that offer the greatest return on investment, the Secretary of Homeland Security should direct the Assistant Secretary for Policy to refine its risk assessment methodology so that in future QHSRs it can compare and prioritize homeland security risks and risk mitigation strategies.

    Agency Affected: Department of Homeland Security

  3. Status: Open

    Comments: As of June 2016, the Office of Policy's Office of Strategy, Plans, Analysis, and Risk finalized a draft stakeholder outreach plan to include use of the Office of Management and Budget's Max electronic collaboration website to engage with federal, state, and local stakeholders. The OMB-MAX website is available to government and non-government offices and allows the posting of documents, articles, and links, as well as facilitating collaborative editing of documents and participant interaction threads, according to DHS officials. In addition, the Office of Policy's Office of Strategy, Plans, Analysis, and Risk is exploring the use of different tools to facilitate more interactive stakeholder engagement. For example, DHS's Office of Partnerships and Engagement is to facilitate additional engagement with external subject matter experts, arrange interagency coordination, and organize review and approval with parties of the homeland security enterprise in order to coordinate and approve the development of the 2018 QHSR. This recommendation will remain open until we verify that interactive communication approaches are implemented.

    Recommendation: To ensure proper management of the QHSR stakeholder consultation process, the Secretary of Homeland Security should direct the Assistant Secretary for Policy to identify and implement stakeholder meeting processes to ensure that communication is interactive when project planning for the next QHSR.

    Agency Affected: Department of Homeland Security

  4. Status: Open

    Comments: As of June 2016, the Office of Policy's Office of Strategy, Plans, Analysis, and Risk (SPAR) drafted a memorandum for the Deputy Secretary to solicit Component subject matter experts. The memorandum specifies component detailee roles and responsibilities, to include serving in an advisory, consultation, and coordination role, according to DHS officials. SPAR is to lead an integrated group of analysts and strategic planners that are to be supported and augmented by the subject matter experts. The experts and detailees are to serve as members of study teams analyzing key threats, trends, and strategy and policy alternatives associated with issues and challenges relating to DHS's mission and objectives. A second memorandum requesting additional detailee support is to be issued in November 2016, prior to the formal review phase of the new QHSR which is to begin in January 2017. This recommendation will remain open until we verify that clarified detailee roles and responsibilities are finalized and implemented.

    Recommendation: To ensure proper management of the internal QHSR stakeholder consultation process, the Secretary of Homeland Security should direct the Assistant Secretary for Policy to clarify component detailee roles and responsibilities when project planning for the next QHSR.

    Agency Affected: Department of Homeland Security

 

Explore the full database of GAO's Open Recommendations »

Oct 17, 2017

Oct 11, 2017

Oct 4, 2017

Sep 27, 2017

Sep 26, 2017

Sep 25, 2017

Sep 12, 2017

Sep 11, 2017

Sep 7, 2017

Looking for more? Browse all our products here