Coast Guard:

Security Risk Model Meets DHS Criteria, but More Training Could Enhance Its Use for Managing Programs and Operations

GAO-12-14: Published: Nov 17, 2011. Publicly Released: Dec 19, 2011.

Additional Materials:

Contact:

Stephen L. Caldwell
(202) 512-9610
contact@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

Since the terrorist attacks of September 11, 2001, the nation's ports and waterways have been viewed as potential targets of attack. The Department of Homeland Security (DHS) has called for using risk-informed approaches to prioritize its investments, and for developing plans and allocating resources that balance security and the flow of commerce. The U.S. Coast Guard--a DHS component and the lead federal agency responsible for maritime security--has used its Maritime Security Risk Analysis Model (MSRAM) as its primary approach for assessing and managing security risks. GAO was asked to examine (1) the extent to which the Coast Guard's risk assessment approach aligns with DHS risk assessment criteria, (2) the extent to which the Coast Guard has used MSRAM to inform maritime security risk decisions, and (3) how the Coast Guard has measured the impact of its maritime security programs on risk in U.S. ports and waterways. GAO analyzed MSRAM's risk assessment methodology and interviewed Coast Guard officials about risk assessment and MSRAM's use across the agency.

MSRAM generally aligns with DHS risk assessment criteria, but additional documentation on key aspects of the model could benefit users of the results. MSRAM generally meets DHS criteria for being complete, reproducible, documented, and defensible. Further, the Coast Guard has taken actions to improve the quality of MSRAM data and to make them more complete and reproducible, including providing training and tools for staff entering data into the model. However, the Coast Guard has not documented and communicated the implications that MSRAM's key assumptions and other sources of uncertainty have on MSRAM's risk results. For example, to assess risk in MSRAM, Coast Guard analysts make judgments regarding such factors as the probability of an attack and the economic and environmental consequences of an attack. These multiple judgments are inherently subjective and constitute sources of uncertainty that have implications that should be documented and communicated to decision makers. Without this documentation, decision makers and external MSRAM reviewers may not have a complete understanding of the uses and limitations of MSRAM data. In addition, greater transparency and documentation of uncertainty and assumptions in MSRAM's risk estimates could also facilitate periodic peer reviews of the model--a best practice in risk management. MSRAM is the Coast Guard's primary tool for managing maritime security risk, but resource and training challenges hinder use of the tool by Coast Guard field operational units, known as sectors. At the national level, MSRAM supports Coast Guard strategic planning efforts, which is consistent with the agency's intent for MSRAM. At the sector level, MSRAM has informed a variety of decisions, but its use has been limited by lack of staff time, the tool's complexity, and competing mission demands, among other things. The Coast Guard has taken actions to address these challenges, but providing additional training on how MSRAM can be used at all levels of sector decision making could further the Coast Guard's risk management efforts. MSRAM is capable of informing operational, tactical, and resource allocation decisions, but the Coast Guard has generally provided MSRAM training only to a small number of sector staff who may not have insight into all levels of sector decision making. The Coast Guard developed an outcome measure to report its performance in reducing maritime risk, but has faced challenges using this measure to inform decisions. Outcome measures describe the intended result of carrying out a program or activity. The measure is partly based on Coast Guard subject matter experts' estimates of the percentage reduction of maritime security risk subject to Coast Guard influence resulting from Coast Guard actions. The Coast Guard has improved the measure to make it more valid and reliable and believes it is a useful proxy measure of performance, noting that developing outcome measures is challenging because of limited historical data on maritime terrorist attacks. However, given the uncertainties in estimating risk reduction, it is unclear if the measure would provide meaningful performance information with which to track progress over time. In addition, the Coast Guard reports the risk reduction measure as a specific estimate rather than as a range of plausible estimates, which is inconsistent with risk analysis criteria. Reporting and using outcome measures that more accurately reflect mission effectiveness can give Coast Guard leaders and Congress a better sense of progress toward goals. GAO recommends that the Coast Guard provide more thorough documentation on MSRAM's assumptions and other sources of uncertainty, make MSRAM available for peer review, implement additional MSRAM training, and report the results of its risk reduction performance measure in a manner consistent with risk analysis criteria. The Coast Guard agreed with these recommendations.

Status Legend:

More Info
  • Review Pending-GAO has not yet assessed implementation status.
  • Open-Actions to satisfy the intent of the recommendation have not been taken or are being planned, or actions that partially satisfy the intent of the recommendation have been taken.
  • Closed-implemented-Actions that satisfy the intent of the recommendation have been taken.
  • Closed-not implemented-While the intent of the recommendation has not been satisfied, time or circumstances have rendered the recommendation invalid.
    • Review Pending
    • Open
    • Closed - implemented
    • Closed - not implemented

    Recommendations for Executive Action

    Recommendation: To help the Coast Guard strengthen MSRAM and better align it with NIPP risk management guidance, as well as facilitate the increased use of MSRAM across the agency, the Commandant of the Coast Guard should provide additional training for sector command staff and others involved in sector management and operations on how MSRAM can be used as a risk management tool to inform sector-level decision making.

    Agency Affected: Department of Homeland Security: United States Coast Guard

    Status: Closed - Implemented

    Comments: In our review of the Coast Guard's Maritime Security Risk Analysis Model (MSRAM), we found that providing additional training on how MSRAM can be used at all levels of sector decision making could further the Coast Guard's risk management efforts. MSRAM is capable of informing operational, tactical, and resource allocation decisions, but the Coast Guard has generally provided MSRAM training only to a small number of sector staff who may not have insight into all levels of sector decision making. In May 2013, Coast Guard reported that MSRAM is currently part of the "Risk-Based Decision Making" lesson taught within the Coast Guard's Contingency Planning Course. The lesson objectives are to identify how risk-based decision making will impact preparedness planning, and demonstrate the uses of MSRAM to develop preparedness plans. The Coast Guard is utilizing other opportunities to provide risk training to Sector command staff, including online and Webinar training opportunities. These actions are consistent with the intent of our recommendation.

    Recommendation: To help the Coast Guard strengthen MSRAM and better align it with NIPP risk management guidance, as well as facilitate the increased use of MSRAM across the agency, the Commandant of the Coast Guard should make MSRAM available to appropriate parties for additional external peer review.

    Agency Affected: Department of Homeland Security: United States Coast Guard

    Status: Closed - Not Implemented

    Comments: In our review of the Coast Guard's Maritime Security Risk Analysis Model (MSRAM), we reported that the Coast Guard was in the process of conducting a verification, validation, and accreditation of MSRAM, and had also been reviewed by risk experts external to the Coast Guard. We noted that as the Coast Guard's risk assessment model continues to evolve, the Coast Guard could benefit from periodic external peer review to ensure that the structure and outputs of the model are appropriate for its given uses and to identify possible areas for improvement. We reported that Coast Guard officials noted they intend to pursue external reviews of MSRAM as part of the ongoing verification, validation, and accreditation (VV&A) process, and, according to Coast Guard, that additional external peer review would be part of an independent verification and validation of MSRAM expected to be completed in the fall of 2012. However, in May 2013 the Coast Guard reported exploring options for additional external peer reviews as part of the MSRAM VV&A process, but determined the extensive scope and cost of the VV&A made it infeasible to incorporate additional external reviews. The Coast Guard added that it did not anticipate conducting additional external reviews of MSRAM in the future. As a result, this recommendation is being closed as not implemented.

    Recommendation: To help the Coast Guard strengthen MSRAM and better align it with NIPP risk management guidance, as well as facilitate the increased use of MSRAM across the agency, the Commandant of the Coast Guard should provide more thorough documentation related to key assumptions and sources of uncertainty within MSRAM and inform users of any implications for interpreting the results from the model.

    Agency Affected: Department of Homeland Security: United States Coast Guard

    Status: Open

    Comments: In our review of the Coast Guard's maritime security risk analysis model (MSRAM), we found that the Coast Guard has not documented and communicated the implications that MSRAM's key assumptions and other sources of uncertainty have on MSRAM's risk results. For example, to assess risk in MSRAM, Coast Guard analysts make judgments regarding such factors as the probability of an attack and the economic and environmental consequences of an attack. These multiple judgments are inherently subjective and constitute sources of uncertainty that have implications that should be documented and communicated to decision makers. Without this documentation, decision makers and external MSRAM reviewers may not have a complete understanding of the uses and limitations of MSRAM data. We recommended that Coast Guard provide more thorough documentation related to key assumptions and sources of uncertainty within MSRAM and inform users of any implications for interpreting the results from the model. In May 2013, Coast Guard reported that documentation of uncertainty continues to be part of the ongoing MSRAM data revalidation process, and documentation of key assumptions and sources of uncertainty was addressed in the ongoing MSRAM Verification, Validation, and Accreditation (VV&A) process. In addition, Coast Guard reported that they plan to incorporate uncertainty estimates as part of the MSRAM risk results by the end of 2014.

    Recommendation: To improve the accuracy of the risk reduction measure for internal and external decision-making, the Commandant of the Coast Guard should take action to report the results of the risk reduction measure as a range rather than a point estimate.

    Agency Affected: Department of Homeland Security: United States Coast Guard

    Status: Closed - Implemented

    Comments: In our report on the Coast Guard's maritime security risk analysis model (MSRAM), we found that the Coast Guard's measure of its performance in reducing maritime risk was reported as a specific estimate rather than as a range of plausible estimates, which is inconsistent with risk analysis criteria. To improve the accuracy of the risk reduction measure for internal and external decision-making, we recommended that Coast Guard take action to report the results of the risk reduction measure as a range rather than a point estimate. In March 2014, the Coast Guard noted that the DHS data reporting system is not configured to allow data entry of risk reduction ranges directly into the system. However, the Coast Guard footnoted the risk reduction ranges for the FY13 outcome measure results in the Coast Guard's FY13 Annual Performance Report. Risk reduction ranges and point estimates were also explained in the FY13 PWCS Risk-Based Performance Model Report dated December 20, 2013. The Coast Guard plans to continue to report the risk reduction ranges and point estimates in these documents in the future. These actions are sufficient to close the recommendation.

    Apr 7, 2014

    Mar 31, 2014

    Mar 28, 2014

    Mar 26, 2014

    Mar 12, 2014

    Mar 7, 2014

    Feb 27, 2014

    Feb 13, 2014

    Looking for more? Browse all our products here