Homeland Security Grant Program:

Additional Actions Could Further Enhance FEMA's Risk-Based Grant Assessment Model

GAO-18-354: Published: Sep 6, 2018. Publicly Released: Sep 6, 2018.

Additional Materials:

Contact:

Chris P. Currie
(202) 512-8777
CurrieC@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

The Federal Emergency Management Agency distributes money to states, territories, and urban areas to help them get ready for potential terrorist attacks. The amounts provided are based in part on a model FEMA uses to rank the risk posed to each area. Locations deemed at higher risk of an attack with major consequences, for example, may receive more money.

We looked at FEMA's main grant programs and what it has done to improve its risk model. We recommended FEMA take additional steps to bolster the model's quality and credibility, including fully documenting the assumptions on which the model is based and subjecting it to external review.

 

This photo shows FEMA headquarters.

This photo shows FEMA headquarters.

Additional Materials:

Contact:

Chris P. Currie
(202) 512-8777
CurrieC@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

What GAO Found

GAO found that various factors affected Federal Emergency Management Agency (FEMA) State Homeland Security Program (SHSP) and Urban Area Security Initiative (UASI) grant awards from fiscal year 2008 through 2018. SHSP grant awards to states were based on two factors—(1) minimum amounts set in law each year, and (2) FEMA's risk model. For example, in fiscal year 2012, each state was to receive a minimum of approximately $2.74 million, with each state receiving additional funds based on its relative risk score. Conversely, UASI grant awards are made based on its FEMA's risk-based grant assessment model, which ranks each urban area relative to others in that year, and Department of Homeland Security (DHS) leadership decisions on how funding should be allocated. From fiscal year 2008 through 2018, the number of USAI grantees varied from year to year (see figure below).

Annual Number of Urban Area Security Initiative (UASI) Grantees for Fiscal Years 2008 through 2018

Annual Number of Urban Area Security Initiative (UASI) Grantees for Fiscal Years 2008 through 2018

Since 2008, FEMA has taken steps to strengthen its risk-based grant assessment model, but has not incorporated additional scientific practices into its model. For example, in 2011 FEMA included more information in its model on potential targets and their vulnerability in each state and urban area, addressing a prior GAO recommendation. More recently in 2018, FEMA added additional factors to better assess vulnerability in each state and urban area, such as the number of special events where large crowds gather and soft targets susceptible to lone wolf attacks, among other things. However, GAO found that FEMA does not fully utilize scientific practices recognized by the National Research Council and the Office of Management and Budget as best practices. Specifically, FEMA did not fully document its model's underlying assumptions, such as the weights in its model or the justification for changes to these weights. FEMA also did not perform the level of analysis needed to determine how changes to its model could affect the resulting risk scores. Finally, FEMA has not coordinated an independent external peer review of its model. Applying such scientific practices could assist FEMA in further strengthening its model.

Why GAO Did This Study

FEMA, a component of DHS, provides preparedness grants to state, local, tribal, and territorial governments to help prepare for, prevent, protect against, respond to, recover from and mitigate terrorist attacks or other disasters. SHSP grants fund the nation's 56 states and territories, while UASI grants fund eligible urban areas. Grant allocations have been based, in part, on FEMA's risk-based grant assessment model, with states and urban areas deemed to be at higher risk receiving more grant dollars than those deemed at lower risk. Since 2008, GAO and others have assessed the model and made recommendations to strengthen it.

This report 1) describes SHSP and UASI grant awards during fiscal years 2008 through 2018, and factors affecting grant distributions; and 2) examines the steps that FEMA has taken to strengthen its risk assessment model for allocating preparedness grants, and any additional opportunities to improve the model. GAO analyzed the information in FEMA's model, and data on SHSP and UASI grant awards for fiscal years 2008 through 2018. GAO also interviewed FEMA and DHS officials and collected documents.

What GAO Recommends

GAO is making three recommendations to FEMA to further strengthen its risk-based grant assessment model by (1) fully documenting the model's assumptions and justifications, (2) performing additional in-depth analyses, and (3) coordinating an external peer review. FEMA concurred with our recommendations.

For more information, contact Chris P. Currie at (202) 512-8777 or 20CurrieC@gao.gov.

Recommendations for Executive Action

  1. Status: Closed - Implemented

    Comments: FEMA agreed that internal and external stakeholders should be provided a justification on why changes were made to the risk methodology as well as the distribution percentage weights for each of the overall components and individual elements. In July 2019, FEMA provided GAO with records of the agency's additional efforts documenting changes to their risk allocation formula. As part of the letter to stakeholders documenting changes made to the data sources or model weights, FEMA provided the reasoning behind those changes. Further, FEMA performed a sensitivity analysis for the Fiscal Year 2019 risk allocation model, and commissioned an external review of the allocation model. According to FEMA, the agency will continue to provide detailed information on the data and risk assessment model makeup as well as reasoning behind any changes to the model and/or data sources. This will continue to provide jurisdictions with the full reasoning behind any changes. In addition, these actions should increase the transparency of the allocation model to the benefit internal decision makers in any future efforts to improve the allocation model's risk methodology. As a result of these actions this recommendation is considered closed as implemented.

    Recommendation: The FEMA Administrator should fully document the underlying assumptions and justifications that form the basis of the risk-based grant assessment model, such as the size of the weights assigned to Threat, Vulnerability, and Consequence, or the justification for changes to these weights from one year to the next.

    Agency Affected: Department of Homeland Security: Directorate of Emergency Preparedness and Response: Federal Emergency Management Agency

  2. Status: Closed - Implemented

    Comments: FEMA concurred with our recommendation, stating that the agency will expand the use of sensitivity analysis to review the entire risk methodology, and will also document these results for leadership review, as appropriate. In July 2019, FEMA provided us with their sensitivity analysis of the Fiscal Year 2019 risk allocation model. According to FEMA, their analysis focused on the sensitivity of computed risk to changes in variable values. Specifically, FEMA's analysis considered each jurisdiction individually, and varied each of the model's variable (e.g. down by 10% of its value, or by one level for discrete inputs). FEMA then compared the impact on the variation on the risk score for each jurisdiction and ranked the variables by the magnitude of this impact. By performing this analysis, FEMA is developing a greater understanding of the how uncertainty affects its risk-based grant assessment model's results. In addition, this action should increase the transparency of the allocation model to the benefit internal decision makers in any future efforts to improve the allocation model's risk methodology. As a result of this action, this recommendation is closed as implemented.

    Recommendation: The FEMA Administrator should perform sensitivity analyses to verify how changes to the risk-based grant assessment model could affect the resulting risk scores, and document the results.

    Agency Affected: Department of Homeland Security: Directorate of Emergency Preparedness and Response: Federal Emergency Management Agency

  3. Status: Closed - Implemented

    Comments: FEMA concurred with our report recommendation, stating that they would coordinate an independent external peer review and develop a detailed written response to leadership for further appropriate action. In July 2019, FEMA transmitted to GAO an external subject matter expert's review of the risk allocation model. This outside expert in the field risk assessment performed an external evaluation of agency's risk model and grant allocation approach for the State Homeland Security Program (SHSP) and Urban Area Security Initiative (UASI) grants. This evaluation included a review of FEMA documents that summarized their allocation model and the history of FEMA's risk assessment process, along with discussions with FEMA officials. By conducting an independent external peer review of the methodology of its risk-based grant assessment model, FEMA bolsters its ability to provide reasonable assurance that the agency's risk model and grant allocation approach are reasonable, of high-quality, and credible. As a result of this action, this recommendation closed as implemented.

    Recommendation: The FEMA Administrator should take steps to coordinate an independent, external peer review of its risk-based grant assessment model.

    Agency Affected: Department of Homeland Security: Directorate of Emergency Preparedness and Response: Federal Emergency Management Agency

 

Explore the full database of GAO's Open Recommendations »

Jul 15, 2020

Jul 14, 2020

Jul 8, 2020

May 14, 2020

May 7, 2020

May 5, 2020

May 4, 2020

May 1, 2020

Looking for more? Browse all our products here