This is the accessible text file for GAO report number GAO-12-14 
entitled 'Coast Guard: Security Risk Model Meets DHS Criteria, but 
More Training Could Enhance Its Use for Managing Programs and 
Operations' which was released on December 19, 2011. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as 
part of a longer term project to improve GAO products' accessibility. 
Every attempt has been made to maintain the structural and data 
integrity of the original printed product. Accessibility features, 
such as text descriptions of tables, consecutively numbered footnotes 
placed at the end of the file, and the text of agency comment letters, 
are provided but may not exactly duplicate the presentation or format 
of the printed version. The portable document format (PDF) file is an 
exact electronic replica of the printed version. We welcome your 
feedback. Please E-mail your comments regarding the contents or 
accessibility features of this document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

United States Government Accountability Office: 
GAO: 

Report to Congressional Requesters: 

November 2011: 

Coast Guard: 

Security Risk Model Meets DHS Criteria, but More Training Could 
Enhance Its Use for Managing Programs and Operations: 

GAO-12-14: 

GAO Highlights: 

Highlights of GAO-12-14, a report to congressional requesters. 

Why GAO Did This Study: 

Since the terrorist attacks of September 11, 2001, the nation’s ports 
and waterways have been viewed as potential targets of attack. The 
Department of Homeland Security (DHS) has called for using risk-
informed approaches to prioritize its investments, and for developing 
plans and allocating resources that balance security and the flow of 
commerce. The U.S. Coast Guard—-a DHS component and the lead federal 
agency responsible for maritime security—-has used its Maritime 
Security Risk Analysis Model (MSRAM) as its primary approach for 
assessing and managing security risks. GAO was asked to examine (1) 
the extent to which the Coast Guard’s risk assessment approach aligns 
with DHS risk assessment criteria, (2) the extent to which the Coast 
Guard has used MSRAM to inform maritime security risk decisions, and 
(3) how the Coast Guard has measured the impact of its maritime 
security programs on risk in U.S. ports and waterways. GAO analyzed 
MSRAM’s risk assessment methodology and interviewed Coast Guard 
officials about risk assessment and MSRAM’s use across the agency. 

What GAO Found: 

MSRAM generally aligns with DHS risk assessment criteria, but 
additional documentation on key aspects of the model could benefit 
users of the results. MSRAM generally meets DHS criteria for being 
complete, reproducible, documented, and defensible. Further, the Coast 
Guard has taken actions to improve the quality of MSRAM data and to 
make them more complete and reproducible, including providing training 
and tools for staff entering data into the model. However, the Coast 
Guard has not documented and communicated the implications that MSRAM’
s key assumptions and other sources of uncertainty have on MSRAM’s 
risk results. For example, to assess risk in MSRAM, Coast Guard 
analysts make judgments regarding such factors as the probability of 
an attack and the economic and environmental consequences of an 
attack. These multiple judgments are inherently subjective and 
constitute sources of uncertainty that have implications that should 
be documented and communicated to decision makers. Without this 
documentation, decision makers and external MSRAM reviewers may not 
have a complete understanding of the uses and limitations of MSRAM 
data. In addition, greater transparency and documentation of 
uncertainty and assumptions in MSRAM’s risk estimates could also 
facilitate periodic peer reviews of the model—a best practice in risk 
management. 

MSRAM is the Coast Guard’s primary tool for managing maritime security 
risk, but resource and training challenges hinder use of the tool by 
Coast Guard field operational units, known as sectors. At the national 
level, MSRAM supports Coast Guard strategic planning efforts, which is 
consistent with the agency’s intent for MSRAM. At the sector level, 
MSRAM has informed a variety of decisions, but its use has been 
limited by lack of staff time, the tool’s complexity, and competing 
mission demands, among other things. The Coast Guard has taken actions 
to address these challenges, but providing additional training on how 
MSRAM can be used at all levels of sector decision making could 
further the Coast Guard’s risk management efforts. MSRAM is capable of 
informing operational, tactical, and resource allocation decisions, 
but the Coast Guard has generally provided MSRAM training only to a 
small number of sector staff who may not have insight into all levels 
of sector decision making. 

The Coast Guard developed an outcome measure to report its performance 
in reducing maritime risk, but has faced challenges using this measure 
to inform decisions. Outcome measures describe the intended result of 
carrying out a program or activity. The measure is partly based on 
Coast Guard subject matter experts’ estimates of the percentage 
reduction of maritime security risk subject to Coast Guard influence 
resulting from Coast Guard actions. The Coast Guard has improved the 
measure to make it more valid and reliable and believes it is a useful 
proxy measure of performance, noting that developing outcome measures 
is challenging because of limited historical data on maritime 
terrorist attacks. However, given the uncertainties in estimating risk 
reduction, it is unclear if the measure would provide meaningful 
performance information with which to track progress over time. In 
addition, the Coast Guard reports the risk reduction measure as a 
specific estimate rather than as a range of plausible estimates, which 
is inconsistent with risk analysis criteria. Reporting and using 
outcome measures that more accurately reflect mission effectiveness 
can give Coast Guard leaders and Congress a better sense of progress 
toward goals. 

What GAO Recommends: 

GAO recommends that the Coast Guard provide more thorough 
documentation on MSRAM’s assumptions and other sources of uncertainty, 
make MSRAM available for peer review, implement additional MSRAM 
training, and report the results of its risk reduction performance 
measure in a manner consistent with risk analysis criteria. The Coast 
Guard agreed with these recommendations. 

View [hyperlink, http://www.gao.gov/products/GAO-12-14]. For more 
information, contact Stephen L. Caldwell at (202) 512-9610 or 
caldwells@gao.gov. 

[End of section] 

Contents: 

Letter: 

Background: 

MSRAM Risk Assessments Generally Align with DHS Criteria, but 
Challenges Remain: 

Coast Guard Has Used a Risk-Informed Approach to Manage Maritime 
Security Risk, but Challenges Hinder Sector Efforts: 

Coast Guard Measures Risk Reduction but Has Faced Challenges Using 
This Measure to Inform Decisions: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments and Our Evaluation: 

Appendix I: Risk Management Framework: 

Appendix II: Comments from the Department of Homeland Security: 

Appendix III: GAO Contact and Staff Acknowledgments: 

Related GAO Products: 

Tables: 

Table 1: Target Types and Attack Modes in MSRAM: 

Table 2: Description of Vulnerability Factors in MSRAM: 

Table 3: Description of Consequence Factors in MSRAM: 

Table 4: National Infrastructure Protection Plan Core Criteria for 
Risk Assessments: 

Figures: 

Figure 1: MSRAM Can Be Used to Inform a Variety of Coast Guard 
Activities and Operations, Including Escorting Ferries, Naval Vessels 
or Cruise Ships, and Waterborne or Aerial Patrols of Critical 
Infrastructure or National Symbols: 

Figure 2: GAO's Risk Management Framework: 

Abbreviations: 

CREATE: National Center for Risk and Economic Analysis of Terrorism 
Events: 

DHS: Department of Homeland Security: 

EGIS: Enterprise Geographic Information System: 

ICC: Intelligence Coordination Center: 

IMPLAN: Impact Analysis for Planning: 

IV&V: independent verification and validation: 

MCIKR: maritime critical infrastructure and key resources: 

MSRAM: Maritime Security Risk Analysis Model: 

MSRO: Maritime Security and Response Operations: 

MTSA: Maritime Transportation Security Act of 2002: 

NIPP: National Infrastructure Protection Plan: 

OMB: Office of Management and Budget: 

PSRAT: Port Security Risk Assessment Tool: 

PWCS: Ports, Waterways, and Coastal Security: 

VV&A: verification, validation, and accreditation: 

[End of section] 

United States Government Accountability Office: 
Washington, DC 20548: 

November 17, 2011: 

The Honorable John D. Rockefeller, IV:
Chairman:
Committee on Commerce, Science, and Transportation:
United States Senate: 

The Honorable Susan M. Collins:
Ranking Member:
Committee on Homeland Security and Governmental Affairs:
United States Senate: 

The Honorable Candice S. Miller:
Chairwoman:
Subcommittee on Border and Maritime Security:
Committee on Homeland Security:
House of Representatives: 

Since the terrorist attacks of September 11, 2001, the nation's ports 
have been viewed as potential targets of attack for many reasons. 
Ports, waterways, and vessels are part of an economic engine handling 
more than $700 billion in merchandise annually, according to the 
Department of Homeland Security (DHS), and an attack on this system 
could have a widespread impact on global shipping, international 
trade, and the U.S. economy. The U.S. Coast Guard--a component of DHS--
is the lead federal agency for maritime security, which includes the 
protection of U.S. ports, coasts, and inland waterways as part of its 
Ports, Waterways, and Coastal Security (PWCS) mission. This mission 
involves protecting the maritime domain and marine transportation 
system, including preventing terrorist attacks, and responding to and 
recovering from attacks that do occur. In addition to its PWCS 
mission, the Coast Guard has 10 other statutory missions.[Footnote 1] 

Since it is not practical or economically feasible to protect all 
assets against every possible terrorist risk, DHS has called for using 
risk-informed approaches to prioritize its investments and for 
developing plans and allocating resources that balance security and 
the flow of commerce. Risk management is a tool for informing 
policymakers' decisions about assessing risks, allocating resources, 
and taking actions under conditions of uncertainty. A risk management 
approach entails a continuous process of managing risk through a 
series of actions, including setting strategic goals and objectives, 
assessing risk, evaluating alternatives, selecting initiatives to 
undertake, and implementing and monitoring those initiatives. DHS 
detailed this approach in its National Infrastructure Protection Plan 
(NIPP), which it issued in June 2006 and updated in 2009.[Footnote 2] 
The 2009 update of the NIPP increased the plan's emphasis on risk 
management, including providing the core criteria of a risk assessment 
approach.[Footnote 3] 

The Coast Guard's primary approach to assessing and managing security 
risks has been embodied in its Maritime Security Risk Analysis Model 
(MSRAM). Since its development and implementation in 2005, MSRAM has 
provided the Coast Guard with a standardized way of assessing risk to 
maritime infrastructure, referred to in MSRAM as targets, which can 
include chemical facilities, oil refineries, hazardous cargo vessels, 
passenger ferries, and cruise ship terminals, to name a few. MSRAM is 
designed to allow comparison between different targets at the local, 
regional, and national levels with the goal of reducing risk by 
prioritizing security activities and resources. MSRAM calculates the 
risk of terrorist attack based on scenarios--a combination of target 
and attack mode--in terms of threats, vulnerabilities, and 
consequences to more than 28,000 maritime targets. For example, a 
MSRAM scenario related to cruise ships could include a boat bomb or an 
attack by a hijacked vessel. 

Since 2004, we have examined Coast Guard efforts to implement a risk 
management framework, noting how the Coast Guard's risk management and 
risk assessment efforts have developed and evolved, as well as how the 
Coast Guard has made progress in assessing maritime security risks 
using MSRAM. For example, in 2005, we reported that by developing 
MSRAM, the Coast Guard had begun to address the limitations of its 
previous port security risk model.[Footnote 4] In 2010, we reported 
that the Coast Guard has strengthened risk management through the 
development and use of MSRAM to help prioritize limited port security 
resources, identify capabilities needed to combat future threats, and 
identify the highest-risk scenarios and targets in the maritime 
domain.[Footnote 5] We also reported in 2010 that the Coast Guard was 
assessing risk to cruise ships and facilities in accordance with DHS 
risk assessment guidance and that the Coast Guard was using MSRAM to 
help concentrate maritime security activities where relative risk is 
believed to be greatest.[Footnote 6] In light of the tight fiscal 
environment combined with the Coast Guard's multiple missions, it is 
critically important for the Coast Guard to make the most effective 
use of its limited resources and to ensure that all levels of the 
Coast Guard are equipped to make risk-informed decisions regarding 
maritime security. 

You asked us to examine the Coast Guard's progress in using MSRAM to 
assess and manage maritime security risk and to assess its progress 
implementing DHS's risk management framework--specifically, how the 
Coast Guard is establishing security priorities based on risk, 
implementing protective programs and strategies, and measuring the 
effectiveness of its actions. This report addresses the following 
questions: 

* To what extent does the Coast Guard's risk assessment approach align 
with DHS risk assessment criteria, and what challenges, if any, exist 
in this effort? 

* To what extent has the Coast Guard used MSRAM to inform maritime 
security risk decisions, and what challenges, if any, exist in this 
effort? 

* How has the Coast Guard measured the impact of its maritime security 
programs on risk in U.S. ports and waterways, and what challenges, if 
any, exist in this effort? 

To address our first objective, we focused on MSRAM, which is the 
Coast Guard's primary model for assessing maritime security risk. We 
compared MSRAM's risk assessment methodology and processes to relevant 
criteria, including the risk assessment component of the 2009 NIPP and 
our related reports on risk management, such as our 2005 report 
examining Coast Guard risk management efforts.[Footnote 7] We 
interviewed Coast Guard headquarters officials to discuss their 
national-level perspectives on the overall management and 
implementation of the MSRAM risk model and we interviewed officials 
from the Coast Guard's Intelligence Coordination Center to discuss how 
intelligence information is incorporated into MSRAM.[Footnote 8] In 
addition, we interviewed officials from all 35 Coast Guard sectors to 
obtain their views on MSRAM. We asked these officials to provide 
information on the MSRAM risk assessment process, MSRAM training, and 
processes and procedures for ensuring MSRAM data integrity. The 
interview was a mix of specific questions using a rating scale and 
questions asking for open-ended or narrative responses. During these 
interviews, we interviewed staff responsible for collecting and 
updating MSRAM data, as well as management-level officials in the 
sectors' response, prevention, planning, and command units. To 
minimize any inconsistencies or errors in the information we 
collected, our subject matter experts developed the interview 
questions in collaboration with a social science survey specialist and 
we pretested the interview with officials from two Coast Guard 
sectors. We provided an advance copy of the interview questions to 
each of the sectors to allow time to prepare responses, conduct 
preliminary research, and identify appropriate points of contact. We 
also validated selected interview response information by 
corroborating it with other sources, such as MSRAM documentation 
provided by the Coast Guard. We conducted these interviews from May 
2011 through August 2011. We also conducted interviews with officials 
from three of the nine Coast Guard districts to obtain their 
perspectives on MSRAM. These districts encompass 15 sectors over the 
West Coast, East Coast, Gulf Coast, and Mississippi River area. Since 
we selected a nonprobability sample of districts, the information 
obtained from these interviews cannot be generalized to all districts 
but provides us with information on how officials from these selected 
districts view MSRAM. In addition, we reviewed external studies on 
MSRAM, including a MSRAM verification and validation report[Footnote 
9] and a report by the National Center for Risk and Economic Analysis 
of Terrorism Events (CREATE) at the University of Southern 
California,[Footnote 10] and we met with the author of the MSRAM 
verification and validation report to discuss the report's findings 
and recommendations. We also reviewed the National Research Council of 
the National Academies study on DHS risk models and spoke with 
individuals involved in conducting that study.[Footnote 11] Although 
this study did not specifically review MSRAM, it provided broad 
information on risk analysis and modeling applicable to MSRAM. We 
reviewed the methodologies of these studies and found them 
sufficiently reliable for the purposes of our report. In addition, we 
met with risk management experts from DHS's Office of Risk Management 
and Analysis to obtain their views on risk analysis. We also met with 
an external risk management expert familiar with MSRAM and Coast Guard 
risk management efforts to gain additional perspective on MSRAM. While 
our review focused on MSRAM, we also obtained information regarding 
other Coast Guard risk management models that are under development to 
see how these models are expected to align with MSRAM. 

To address our second objective, we also focused on MSRAM because it 
is the Coast Guard's primary maritime security risk management tool. 
We reviewed Coast Guard documents describing MSRAM's current and 
intended uses at headquarters and in the field. We also relied on the 
interviews with officials from the 35 Coast Guard sectors and the 3 
Coast Guard districts as previously described. During these 
interviews, we obtained information on how sectors and districts use 
MSRAM at the local level to guide tactical, operational, and strategic 
security efforts and strengths and limitations of MSRAM for these 
purposes. Based on the information provided by these interviews, we 
compared sectors' reported uses of MSRAM for informing risk-based 
decision making with Coast Guard documentation on the intended uses of 
MSRAM, including Commandant Instructions and internal Coast Guard risk 
management guidance documents. We verified the intended uses of MSRAM 
with officials from Coast Guard headquarters. At Coast Guard 
headquarters, we interviewed personnel from Coast Guard offices that 
use MSRAM information to support decision making, as well as officials 
responsible for developing risk assessment training standards within 
the Coast Guard. 

To address our third objective, we focused on the Coast Guard's risk 
reduction performance measure and its supporting model. This measure 
is the Coast Guard's primary method for measuring and reporting its 
overall performance in reducing risk in the maritime domain. We 
reviewed Coast Guard documentation on the risk reduction measure and 
supporting model, as well as relevant criteria, including our criteria 
for performance measurement,[Footnote 12] the NIPP framework, and the 
Office of Management and Budget (OMB) Updated Principles for Risk 
Analysis.[Footnote 13] Additionally, we interviewed Coast Guard 
officials to discuss how the risk reduction measure and supporting 
model are used as well as recent and planned improvements. We also 
discussed the Coast Guard's risk reduction measure with a senior DHS 
official responsible for reviewing department and component-level 
performance measures. 

We conducted this performance audit from October 2010 through November 
2011 in accordance with generally accepted government auditing 
standards. Those standards require that we plan and perform the audit 
to obtain sufficient, appropriate evidence to provide a reasonable 
basis for our findings and conclusions based on our audit objectives. 
We believe that the evidence obtained provides a reasonable basis for 
our findings and conclusions based on our audit objectives. 

Background: 

Risk Management: 

In recent years, we, Congress, the 9/11 Commission, and others have 
recommended that federal agencies with homeland security 
responsibilities utilize a risk management approach to help ensure 
that finite resources are dedicated to assets or activities considered 
to have the highest security priority. The purpose of risk management 
is not to eliminate all risks, as that is an impossible task. Rather, 
given limited resources, risk management is a structured means of 
making informed trade-offs and choices about how to use available 
resources effectively and monitoring the effect of those choices. 
Thus, risk management is a continuous process that includes the 
assessment of threats, vulnerabilities, and consequences to determine 
what actions should be taken to reduce or eliminate one or more of 
these elements of risk. 

To provide guidance to agency decision makers, we developed a risk 
management framework, which is intended to be a starting point for 
applying risk-informed principles. Our risk management framework 
entails a continuous process of managing risk through a series of 
actions, including setting strategic goals and objectives, assessing 
risk, evaluating alternatives, selecting initiatives to undertake, and 
implementing and monitoring those initiatives. Additional information 
on risk management, including our risk management framework, can be 
found in appendix I. 

DHS is required by statute to utilize risk management principles with 
respect to various DHS functions.[Footnote 14] With regard to the 
Coast Guard, federal statutes call for the Coast Guard to use risk 
management in specific aspects of its homeland security efforts. The 
Maritime Transportation Security Act of 2002 (MTSA), for example, 
calls for the Coast Guard and other port security stakeholders, 
through implementing regulations, to carry out certain risk-based 
tasks, including assessing risks and developing security plans for 
ports, facilities, and vessels.[Footnote 15] In addition, the Coast 
Guard Authorization Act of 2010 requires, for example, the Coast Guard 
to (1) develop and utilize a national standard and formula for 
prioritizing and addressing assessed security risks at U.S. ports and 
facilities, such as MSRAM; (2) require Area Maritime Security 
Committees[Footnote 16] to use this standard to regularly evaluate 
each port's assessed risk and prioritize how to mitigate the most 
significant risks; and (3) make MSRAM available, in an unclassified 
version, on a limited basis to regulated vessels and facilities to 
conduct risk assessments of their own facilities and vessels.[Footnote 
17] 

Coast Guard Security Risk Models: 

From 2001 to 2006, the Coast Guard assessed maritime security risk 
using the Port Security Risk Assessment Tool (PSRAT), which was 
quickly developed and fielded after the terrorist attacks of September 
11, 2001. PSRAT served as a rudimentary risk calculator that ranked 
maritime critical infrastructure and key resources (MCIKR) with 
respect to the consequences of a terrorist attack and evaluated 
vessels and facilities that posed a high risk of a transportation 
security incident. While PSRAT provided a relative risk of targets 
within a port region, it could not compare and prioritize relative 
risks of various infrastructures across ports, among other limitations. 

Recognizing the shortcomings of PSRAT that had been identified by the 
Coast Guard and us, in 2005 the Coast Guard developed and implemented 
MSRAM to provide a more robust and defensible terrorism risk analysis 
process. MSRAM is a risk-based decision support tool designed to help 
the Coast Guard assess and manage maritime security risks throughout 
the Coast Guard's area of responsibility. Coast Guard units throughout 
the country use this tool to assess security risks to over 28,000 key 
maritime infrastructure assets--also known as targets--such as 
chemical facilities, passenger terminals, and bridges, as well as 
vessels such as cruise ships, ferries, and vessels carrying hazardous 
cargoes, among other things.[Footnote 18] Unlike PSRAT, MSRAM is 
designed to capture the security risks facing different types of 
targets, allowing comparison between different targets and geographic 
areas at the local, regional, and national levels. MSRAM's risk 
assessment methodology assesses the risk of a terrorist attack based 
on different scenarios; that is, it combines potential targets with 
different attack modes for each target/attack mode combination (see 
table 1). MSRAM automatically determines which attack modes are 
required to be assessed for each target type, though local MSRAM 
analysts have the ability to evaluate additional optional attack modes 
against any target.[Footnote 19] For each target/attack mode 
combination, MSRAM can provide different risk results, such as the 
inherent risk of a target and the amount of risk mitigated by Coast 
Guard security efforts. 

Table 1: Target Types and Attack Modes in MSRAM: 

Examples of target types: 
* MTSA-regulated facilities and offshore platforms; 
* Domestic/foreign barges carrying certain dangerous cargoes; 
* Certain vessels; 
* Key infrastructure (e.g., pipelines, bridges, and tunnels); 
* Key assets (e.g., nuclear power plants and dams); 
* Non-regulated high consequence targets that are critical to port 
operations (e.g., non-maritime bridges feeding the port area); 
* Non-regulated high consequence targets that are plausibly attackable 
from boat bombs, such as high-rise buildings located directly on water; 
* Special events and waterside attractions; 
* Targets involved in military outloads. 

Examples of attack modes: 
* Boat bomb; 
* Truck bomb; 
* Small suicide aircraft; 
* Swimmer/diver/underwater delivery systems; 
* Passenger/passerby explosives/improvised explosive device; 
* Mines (aquatic); 
* Attack by hijacked vessel; 
* Attack by hijacked large aircraft; 
* Attack by terrorist assault team; 
* Sabotage; 
* Boat bomb (while vessel is present); 
* Multiple boat attack. 

Source: U.S. Coast Guard. 

[End of table] 

MSRAM calculates risk using the following risk equation: Risk = Threat 
x Vulnerability x Consequence. Numerical values representing Coast 
Guard's assessment of threat (or relative likelihood of attack), 
vulnerability should an attack occur, and consequences of a successful 
attack are combined to yield a risk score for each maritime target. 
The model calculates risk using threat judgments provided by the Coast 
Guard Intelligence Coordination Center (ICC), and vulnerability and 
consequence judgments provided by MSRAM users at the sector level--
typically Coast Guard port security specialists--which are reviewed at 
the district, area, and headquarters levels. The risk equation 
variables are as follows: 

Threat: 

Threat represents the relative likelihood of an attempted attack on a 
target. The ICC provides threat probabilities to MSRAM, based upon 
judgments regarding specific intent, capability, and geographic 
preference of terrorist organizations to deliver an attack on a 
specific type of maritime target class--for example, a boat bomb 
attack on a ferry terminal. To make these judgments, ICC officials use 
intelligence reports generated throughout the broader intelligence 
community to make qualitative determinations about certain terrorist 
organizations and the threat they pose to the maritime domain. At the 
sector level, Coast Guard MSRAM users do not input threat 
probabilities and are required to use the threat probabilities 
provided by the ICC. This approach is intended to ensure that threat 
information is consistently applied across ports. 

Vulnerability: 

Vulnerability represents the probability of a successful attack given 
an attempt. MSRAM users at the sector level assess the vulnerability 
of targets within their respective areas of responsibility. Table 2 
shows the factors included in the MSRAM vulnerability assessment. 

Table 2: Description of Vulnerability Factors in MSRAM: 

Vulnerability factors: Achievability; 
Definitions: A measure of the ability to successfully attack the 
target in the absence of security measures. This factor is designed to 
capture the innate degree of difficulty of the attack on a target. For 
example, weather or climate requirements for the scenario (wind, 
temperature, etc.) may alter the potential likelihood of the attack. 

Vulnerability factors: System security; 
Definitions: A measure of the probability that the security strategy 
in place, made up of the owner/operator, law enforcement agencies, or 
the Coast Guard, will successfully interdict a terrorist attack before 
it occurs. 

Vulnerability factors: Target hardness; 
Definitions: A measure of the target's ability to physically withstand 
the specific attack type. 

Source: U.S. Coast Guard. 

[End of table] 

Consequence: 

Consequence represents the projected overall impact of a successful 
attack on a given target or asset.[Footnote 20] Similar to 
vulnerability assessments, MSRAM users at the sector level assess the 
consequences of a successful attack on targets within their respective 
area of responsibility. Table 3 shows the factors included in the 
MSRAM consequence assessment. 

Table 3: Description of Consequence Factors in MSRAM: 

Consequence factors: Death/injury; 
Definitions: Represents the expected number of deaths/injuries from a 
successful attack. This includes both deaths at the time of attack, 
and deaths that occur later but are still clearly a direct result of 
the attack (e.g., burn victims, or victims who become sick and die 
from exposure to chemical or biological agents). 

Consequence factors: Economic - primary; 
Definitions: Represents the expected property damage and immediate 
business interruption from a successful attack. This includes the 
actual costs of replacing or repairing maritime infrastructure, as 
well as business losses resulting from the attack. 

Consequence factors: Environmental; 
Definitions: Represents the expected environmental impacts of a 
successful attack. This impact predominately captures impacts from oil 
and oil-like substances. 

Consequence factors: National security; 
Definitions: Represents the expected impact of a successful attack on 
a target involved in providing national security. 

Consequence factors: Symbolic; 
Definitions: Represents the symbolic impact of a successful attack 
based on the iconic value of the target in terms of its local, 
regional, national, and international importance. 

Consequence factors: Economic - secondary; 
Definitions: Represents the expected follow-on economic effects of a 
successful attack. For example, an attack on a fuel refinery could 
interrupt energy production and distribution, which is considered a 
secondary economic effect. This assessment should take into account 
redundancy and recoverability of the target. 

Source: U.S. Coast Guard. 

[End of table] 

In addition to the consequence factors listed in table 3, sector MSRAM 
users also assess the response capabilities of the Coast Guard, port 
stakeholders, and other governmental agencies and their ability to 
mitigate death/injury, primary economic, and environmental 
consequences of a successful attack. Because there is a broad array of 
target types operating in the maritime domain that can result in 
different types of impacts if successfully attacked, MSRAM uses an 
approach for drawing equivalencies between the different types of 
impacts. This approach was based on establishing a common unit of 
measure, called a consequence point. One consequence point represents 
$1 million of equivalent loss to the American public.[Footnote 21] 

To support MSRAM development and risk analysis at the headquarters 
level, the Coast Guard has provided MSRAM-dedicated staff and 
resources. According to the Coast Guard, resources for MSRAM or port 
security risk analysis are not from a specific budget line item. From 
fiscal year 2006 to fiscal year 2011, the Coast Guard reported 
assigning from two to five staff (full-time equivalents) and from $0.6 
million to $1.0 million annually to support MSRAM at headquarters. 
There are no MSRAM-dedicated staff at the area, district, and sector 
levels; rather, MSRAM assessment and analysis is generally conducted 
by port security specialists, who have other responsibilities. The 
port security specialist typically has responsibility for numerous 
activities, including the Port Security Grant Program, Area Maritime 
Security Committees, and Area Maritime Security Training Exercise 
Program, among others. 

DHS Risk Management Criteria: 

The NIPP is DHS's primary guidance document for conducting risk 
assessments and includes core criteria that identify the 
characteristics and information needed to produce quality risk 
assessment results. The NIPP's basic analytical principles state that 
risk assessments should be complete, reproducible, documented, and 
defensible, as defined in table 4. 

Table 4: National Infrastructure Protection Plan Core Criteria for 
Risk Assessments: 

Criterion: Complete; 
Description: The methodology should assess consequence, vulnerability, 
and threat for every defined risk scenario and follow the more 
specific guidance given in NIPP, such as documenting the scenarios 
assessed, estimating the number of fatalities, describing all 
protective measures in place, and identifying attack methods that may 
be employed. 

Criterion: Reproducible; 
Description: The methodology must produce comparable, repeatable 
results, even though assessments of different critical infrastructure 
and key resources may be performed by different analysts or teams of 
analysts. It must minimize the number and impact of subjective 
judgments, leaving policy and value judgments to be applied by 
decision makers. 

Criterion: Documented; 
Description: The methodology and the assessment must clearly document 
what information is used and how it is synthesized to generate a risk 
estimate. Any assumptions, weighting factors, and subjective judgments 
need to be transparent to the user of the methodology, its audience, 
and others who are expected to use the results. The types of decisions 
that the risk assessment is designed to support and the timeframe of 
the assessment (e.g., current conditions versus future operations) 
should be given. 

Criterion: Defensible; 
Description: The risk methodology must logically integrate its 
components, making appropriate use of the professional disciplines 
relevant to the analysis, and be free from significant errors or 
omissions. Uncertainty associated with consequence estimates and 
confidence in the vulnerability and threat estimates should be 
communicated. 

Source: NIPP. 

[End of table] 

MSRAM Risk Assessments Generally Align with DHS Criteria, but 
Challenges Remain: 

MSRAM generally aligns with DHS's criteria for a complete and 
reproducible risk assessment, but some challenges remain, such as the 
limited time for Coast Guard personnel to complete assessments. MSRAM 
also generally aligns with the NIPP criteria for a documented and 
defensible risk assessment, but the Coast Guard could improve its 
documentation of the model's assumptions and other sources of 
uncertainty, such as the subjective judgments made by Coast Guard 
analysts about vulnerabilities and consequences, and how these 
assumptions and other sources of uncertainty affect MSRAM's results. 
In addition to providing decision makers with an understanding of how 
to interpret any uncertainty in MSRAM's risk estimates, greater 
transparency and documentation could facilitate periodic peer reviews 
of the model--a best practice in risk management. 

MSRAM Generally Aligns With NIPP Criteria For A Complete And 
Reproducible Risk Assessment: 

MSRAM generally aligns with NIPP criteria for a complete risk 
assessment. In accordance with NIPP criteria for a complete risk 
assessment, MSRAM assesses risk using three main variables--
consequence, vulnerability, and threat. MSRAM's risk assessment 
methodology also follows the NIPP criteria for factors that should be 
assessed in each of the three risk variables. Specifically, for 
threat, MSRAM generally follows the NIPP criteria by identifying 
attack methods that may be employed and by considering the adversary's 
intent and capability to attack a target. MSRAM generally follows the 
vulnerability assessment criteria by estimating the likelihood of an 
adversary's success for each attack scenario and describing the 
protective measures in place, and MSRAM generally follows the 
consequence assessment criteria by estimating economic loss in 
dollars, estimating fatalities, and describing psychological impacts, 
among other things. 

MSRAM's risk assessment methodology also generally aligns with the 
NIPP criteria for a reproducible risk assessment. To be reproducible, 
the methodology must produce comparable, repeatable results and 
minimize the number and impact of subjective judgments, among other 
things. Although Coast Guard officials acknowledge that MSRAM risk 
data are inherently subjective, the MSRAM model and data collection 
processes include features designed to produce comparable, repeatable 
results across sectors. For instance, the Coast Guard prepopulates 
threat data into MSRAM from the Coast Guard's ICC. This allows for 
nationally vetted threat scores that do not rely on multiple 
subjective local judgments. DHS, in its 2010 Transportation Systems 
Sector-Specific Plan, stated that MSRAM produces comparable, 
repeatable results.[Footnote 22] 

Coast Guard Efforts That Contribute to MSRAM Being Complete and 
Reproducible: 

The Coast Guard has taken numerous actions that contribute to MSRAM 
being a complete and reproducible risk assessment model. To improve 
the quality and accuracy of MSRAM data and reduce the amount of 
subjectivity in the MSRAM process, the Coast Guard conducts an annual 
review and validation of MSRAM data produced at each sector; provides 
MSRAM users with tools, calculators, and benchmarks to assist in 
calculating consequence and vulnerability; and provides training to 
sectors on how to enter data into MSRAM. Specific actions are detailed 
below. 

Annual validation and review. The Coast Guard uses a multilevel annual 
validation and review process, which helps to ensure that MSRAM risk 
data are comparable and repeatable across sectors. According to a 2010 
review of MSRAM, conducting a thorough review process across sectors 
is especially important if the data are to be used for national-level 
decision making.[Footnote 23] This process includes sector, district, 
area, and headquarters officials and aims to normalize MSRAM data by 
establishing national averages of risk scores for attack modes and 
targets and by identifying outliers.[Footnote 24] The annual MSRAM 
validation and review process begins with sectors completing 
vulnerability and consequence assessments for targets within their 
areas of responsibility.[Footnote 25] Once the sector Captain of the 
Port validates the assessments, the risk assessment data are sent to 
district and area officials for review.[Footnote 26] Following these 
reviews, Coast Guard headquarters officials combine each sector's data 
into a national classified dataset and perform a statistical analysis 
of the data. The statistical analysis involves calculating national 
averages for vulnerability, consequence, and response capabilities 
risk scores.[Footnote 27] When determining whether a sector's risk 
score for a specific target is questionable or is an outlier, 
reviewers consider the results of the statistical analysis as well as 
supporting comments or rationale provided by sector officials. 

According to the Coast Guard, for each outlier identified during the 
national review process, sector officials reconsider the data point in 
question and either change the inputs to reflect national averages or 
provide additional justification for why the risk score for the target 
in question should be outside of the national average. Headquarters 
officials explained that they generally accept justification for data 
outliers and that a goal of the review process is to spur discussions 
related to maritime risk rather than forcing compliance with national 
data averages. For example, officials from one sector told us that a 
small port in their sector is critical for their state's energy 
imports, and accordingly, the port infrastructure is high risk on a 
national scale. The officials said that Coast Guard headquarters 
officials have questioned the relatively high risk rankings of the 
port's infrastructure because they are statistical outliers, but have 
deferred to the expertise of the sector regarding the risk scores. 

Tools and calculators. Recognizing that sector port security 
specialists who assess risk using MSRAM generally do not have 
expertise in all aspects of assessing vulnerability and consequence, 
the Coast Guard has regularly added new tools and calculators to MSRAM 
to improve the quality, accuracy, and consistency of vulnerability and 
consequence assessments. For example, MSRAM now includes a blast 
calculator that allows users to more easily determine the death and 
injury consequences of an explosion close to population centers. Coast 
Guard officials from 29 sectors (82 percent of sectors) cited a 
variety of challenges with assessing vulnerability and consequence 
values in MSRAM, but officials from 10 sectors said that it was 
becoming easier to do over time and officials from 14 sectors said 
that the tools and calculators in MSRAM have helped. 

Benchmarks and recommended ranges. To limit inconsistencies caused by 
different judgments by individual MSRAM users and to minimize user 
subjectivity, the Coast Guard built into MSRAM a suggested range of 
scores for each risk factor--including vulnerability, consequence, and 
response capabilities--as well as averages, or benchmarks, of scores 
for each factor. The benchmarks are based on Coast Guard and expert 
evaluation of target classes and attack modes. The benchmarks and 
recommended ranges are reviewed and updated each year following the 
annual data revalidation cycle. 

Training. The Coast Guard has also provided annual training for MSRAM 
users, including beginning, intermediate, and advanced courses 
intended to standardize the data entry process across the Coast Guard. 
Officials from 34 sectors (97 percent) reported finding the training 
moderately to very useful in terms of enhancing their ability to 
assess, understand, and communicate the risks facing their sectors. In 
2011, Coast Guard headquarters also started providing live web-based 
training sessions on various MSRAM issues, such as resolving national 
review comments, to help sector staff gain familiarity with MSRAM's 
features on an as-needed basis. In addition to MSRAM training provided 
by headquarters, one Coast Guard district official we spoke with had 
developed and provided localized training to the sector-level port 
security specialists on assessing the vulnerability of chemical 
facilities. The district official told us that Coast Guard 
headquarters was interested in this local model for delivering 
training and was planning to pilot a similar training program in a 
different district. 

Remaining Challenges and Limitations to Making MSRAM Complete and 
Reproducible: 

MSRAM generally aligns with DHS's criteria for a complete and 
reproducible risk assessment, but challenges remain with the MSRAM 
methodology and risk assessment process. The Coast Guard has 
acknowledged these challenges and limitations and has actions underway 
to address them and make MSRAM more complete and reproducible. Coast 
Guard officials noted that some of these challenges are not unique to 
MSRAM and are faced by others in the homeland security risk assessment 
community. Specific challenges are detailed below. 

Data subjectivity. While the Coast Guard has taken actions to minimize 
the subjectivity of MSRAM data, officials acknowledged that assessing 
threat, vulnerability, and consequence is inherently subjective. To 
assess threat, the Coast Guard's ICC quantifies judgments related to 
the intent and capability of terrorist organizations to attack 
domestic maritime infrastructure. However, there are limited national 
historic data for domestic maritime attacks and thus intelligence 
officials must make a number of subjective judgments and draw 
inferences from international maritime attacks. Further, GAO has 
previously reported on the inherently difficult nature of assessing 
the capability and intent of terrorist groups.[Footnote 28] 
Vulnerability and consequence assessments in MSRAM are also inherently 
subjective. For example, officials from 20 sectors we interviewed said 
that even with training, tools, and calculators, assessing 
consequences can be challenging and that it often involved 
subjectivity and uncertainty. Officials noted that assessing economic 
impacts--both primary and secondary--was particularly challenging 
because it required some level of expertise in economics--such as 
supply chains and industry recoverability--which port security 
specialists said is often beyond their skills and training.[Footnote 
29] The input for secondary economic impacts can have a substantial 
effect on how MSRAM's output ranks a target relative to other 
potential targets. Undervaluing secondary economic impacts could 
result in a lower relative risk ranking that underestimates the 
security risk to a target, or inversely, overvaluing secondary 
economic impacts could result in overestimating the security risk to a 
target. Recognizing the challenges with assessing secondary economic 
impacts, Coast Guard officials said they are working with the DHS 
Office of Risk Management and Analysis to study ways to more 
accurately assess secondary economic impacts. Additionally, during the 
course of our review the Coast Guard implemented a tool called IMPLAN 
that has the potential to inform judgments of secondary economic 
impacts by showing what the impact could be for different terrorist 
scenarios.[Footnote 30] 

Limited time to complete assessments. Officials from 19 sectors (54 
percent) told us that the lack of time to complete their annually 
required vulnerability and consequence assessments is a key challenge 
and many expressed that they believed their sector's data suffered in 
quality as a result. Each year, sectors are required to update and 
validate their risk assessments for targets in their areas of 
responsibility, which can involve site visits to port facilities and 
discussions with facility security officers to obtain information on 
vulnerability and consequences. Officials from a Gulf Coast sector 
noted that obtaining this information from facilities can be 
challenging because of the number of facilities in the sector and the 
time involved in meeting with each facility. Officials from an inland 
river sector also noted that gathering data from certain facilities--
such as information on a chemical plant's security enhancements or the 
expected loss of life from a terrorist attack--is challenging because 
facilities may not want to share proprietary information that could be 
damaging in the hands of a competitor. As a result, it often takes 
additional visits, phone calls, e-mails, and time to obtain this 
information. Officials from a northeastern sector said that having the 
people and time to update MSRAM data is their key challenge and 
completing the update is a heavy lift because the update is required 
at the same time as several other requirements, such as reviewing 
investment justifications for the Port Security Grant Program. Coast 
Guard sector officials and one district official we spoke with 
reported raising concerns to headquarters about the time it takes to 
complete MSRAM assessments. Headquarters staff also said they were 
looking into additional ways to make the assessment process easier for 
sectors, such as providing job aids and examining the possibility of 
completing the data update at different times in the year. 

Limitations in modeling methodology--adaptive terrorist behavior. 
There are inherent limitations in the overall methodology the Coast 
Guard uses to model risk. For instance, MSRAM threat information does 
not account for adaptive terrorist behavior, which is defined by the 
National Research Council as an adversary adapting to the perceived 
defenses around targets and redirecting attacks to achieve its goals. 
[Footnote 31] Accounting for adaptive terrorist behavior could be 
modeled by making threat a function of vulnerability and consequence 
rather than the MSRAM formula which treats threat, vulnerability, and 
consequence as independent variables.[Footnote 32] Not accounting for 
adaptive terrorist behavior is a critique of MSRAM raised by terrorism 
risk assessment experts. For example, officials from the DHS Office of 
Risk Management and Analysis have stressed the need to account for 
adaptive terrorist behavior in risk models. In addition, DHS's 2011 
Risk Management Fundamentals guidance states that analysts should be 
careful when calculating risk by multiplying threats, vulnerabilities, 
and consequences (as MSRAM does), especially for terrorism, because of 
interdependencies between the three variables.[Footnote 33] Coast 
Guard officials agreed with the importance of accounting for adaptive 
terrorist behavior and with the risks of treating threat, 
vulnerability, and consequence as independent variables. The officials 
explained that although they did not design MSRAM to account for 
adaptive terrorist behavior, they are working to develop the Dynamic 
Risk Management Model, which will potentially address this issue. 
[Footnote 34] 

Limitations in modeling methodology--network effects. Understanding 
the intent and capabilities of an intelligent adversary is also 
critical for understanding network effects. Network effects involve 
the ripple effect of an incident or simultaneous incidents on key 
sectors of the economy. Assessing network effects could involve 
determining whether a terrorist attack on a few key assets would have 
a disproportionate effect on the performance of the network. A 
starting point for understanding network effects involves gaining a 
greater understanding of how a network is vulnerable to a diverse 
range of threats. Examining how such vulnerabilities create strategic 
opportunities for intelligent adversaries with malevolent intent is 
central to this understanding.[Footnote 35] MSRAM does not assess 
network effects because, according to Coast Guard officials, these 
types of assessments are beyond the intended use of MSRAM. The 2009 
NIPP, the 2010 DHS Quadrennial Review,[Footnote 36] and the National 
Academies[Footnote 37] have determined that gaining a better 
understanding of network effects would help to understand multiplying 
consequences of a terrorist attack or simultaneous attacks on key 
facilities. Although MSRAM does not assess network effects, officials 
from four sectors said they had undertaken local initiatives to 
identify and document networked systems of targets that if 
successfully attacked would have large ripple effects throughout the 
port or local economy. Coast Guard officials agreed that assessing 
network effects is a challenge and they are examining ways to meet 
this challenge.[Footnote 38] However, the Coast Guard's work in this 
area is still in its infancy and there is uncertainty regarding the 
way in which the agency will move forward in measuring network effects. 

Additional Documentation Could Align MSRAM With NIPP Criteria For 
Documented And Defensible Risk Assessments: 

MSRAM is generally documented and defensible, but the Coast Guard 
could improve its documentation of the model's assumptions and other 
sources of uncertainty, such as subjective judgments made by Coast 
Guard analysts about threats, vulnerabilities, and consequences, and 
how these assumptions and other sources of uncertainty affect MSRAM's 
results. The NIPP states that for a risk assessment methodology to be 
documented, any assumptions and subjective judgments need to be 
transparent to the individuals who are expected to use the results. 
For a risk assessment methodology to be defensible, uncertainty 
associated with consequence estimates and the level of confidence in 
the vulnerability and threat estimates should also be communicated to 
users of the results. 

There are multiple assumptions and other sources of uncertainty in 
MSRAM. For example, assumptions used in MSRAM include the particular 
dollar value for a statistical life or the assumed dollar amount of 
environmental damage resulting from oil or hazardous material spilled 
as the result of a terrorist attack. MSRAM also relies on multiple 
subjective judgments made by Coast Guard analysts, which mean a range 
of possible values for risk calculated from the model. For example, to 
assess risk in MSRAM, Coast Guard analysts make judgments regarding 
such factors as the likelihood of success in interdicting an attack 
and the number of casualties expected to result from an attack. These 
subjective judgments are sources of uncertainty with implications 
that, according to the NIPP and risk management best practices, should 
be documented and communicated to decision makers.[Footnote 39] 

MSRAM's primary sources of documentation provide information on how 
data are used to generate a risk estimate and information on some 
assumptions, and the Coast Guard has made efforts to document and 
reduce the number of assumptions made by the field-level user in order 
to increase the consistency of MSRAM's data. For example, the MSRAM 
training and software manual states that MSRAM users are expected to 
specify the assumptions they make in evaluating various attack modes 
and provides assumptions for users to consider when scoring attack 
scenarios, such as specifying the type and amount of biological agent 
used in a biological attack scenario and assuming that attackers are 
armed and suicidal in a boat bomb attack scenario. 

While these documentation efforts are positive steps to reduce MSRAM 
data subjectivity and increase data consistency, we found that the 
Coast Guard has not documented all the sources of uncertainty 
associated with threat, vulnerability, and consequence assessments and 
what implications this uncertainty has for interpreting the results, 
such as an identification of the highest-risk targets in a port. As a 
result, decision makers do not know how robust the risk rankings of 
targets are and the degree to which a list of high-risk targets could 
change given the uncertainty in the risk model's inputs and 
parameters. Moreover, overlapping ranges of possible risk values 
caused by uncertainty could have implications for strategic decisions 
or resource allocation, such as allocating grant funding or targeting 
patrols. Overlapping ranges of risk values due to uncertainty also 
underscores the importance of professional judgment in decision making 
because risk models do not produce precise outcomes that should be 
followed without a degree of judgment and expertise. 

According to the NIPP, the best way to communicate uncertainty will 
depend on the factors that make the outcome uncertain, as well as the 
amount and type of information that is available. The NIPP states that 
in any given terrorist attack scenario there is often a range of 
outcomes that could occur, such as a range of dollar amounts for 
environmental damage or a range of values for a statistical life. For 
some incidents, the range of outcomes is small and a single estimate 
may provide sufficient data to inform decisions. However, if the range 
of outcomes is large, the scenario may require additional specificity 
about conditions to obtain appropriate estimates of the outcomes. 
Often, this means providing a range of possible outcomes rather than a 
single point estimate. Coast Guard officials agreed with the 
importance of documenting and communicating the sources and 
implications of uncertainty for MSRAM's risk estimates, and noted that 
they planned to develop this documentation as part of an internal 
MSRAM verification, validation, and accreditation (VV&A) process that 
they expect to complete in the fall of 2011.[Footnote 40] According to 
the Coast Guard, accreditation is an official determination that a 
model or simulation is acceptable to use for a specific purpose. While 
this accreditation process is expected to document the scope and 
limitations of MSRAM's capabilities and determine whether these 
capabilities are appropriate for MSRAM's current use, the Coast 
Guard's draft accreditation plan does not discuss how the Coast Guard 
plans to assess and document uncertainty in its model or communicate 
those results to decision makers. 

In addition to providing decision makers with an understanding of how 
to interpret uncertainty in MSRAM's risk estimates, greater 
transparency and documentation of uncertainty and assumptions could 
also facilitate periodic peer reviews of the model--a best practice in 
risk management. According to the National Research Council of the 
National Academies, periodic reviews and evaluations of risk model 
outputs are important for transparency with respect to decision 
makers.[Footnote 41] Further, these reviews should involve specialists 
in modeling and in the problems that are being addressed and should 
address the structure of the model, the types and certainty of the 
data, and how the model is intended to be used. Peer reviews can also 
identify areas for improvement and can facilitate sharing best 
practices. As we have previously reported, external peer reviews 
cannot ensure the success of a model, but they can increase the 
probability of success by improving the technical quality of projects 
and the credibility of the decision-making process.[Footnote 42] MSRAM 
has been reviewed twice--in 2010 by risk experts affiliated with the 
Naval Postgraduate School and, to a lesser extent, in 2009 by CREATE 
at the University of Southern California. The authors of the Naval 
Postgraduate School report stated that their review was intended to 
validate and verify the equations used in MSRAM, evaluate MSRAM's 
quality control procedures, and review the use of MSRAM outputs to 
manage risk.[Footnote 43] The authors of the CREATE report stated that 
their review focused on suggestions for improvement rather than a 
comprehensive evaluation, and they suggested that the Coast Guard 
continue to seek feedback and reviews from the risk and decision 
analysis community, as well as from practitioners of other 
disciplines.[Footnote 44] Coast Guard officials told us that they have 
generally benefited from reviews of MSRAM and have worked to implement 
many of the resulting recommendations. Officials noted they intend to 
pursue external reviews of MSRAM as part of the ongoing VV&A process, 
but they have not identified who would be conducting the reviews, or 
when the reviews would occur. As the Coast Guard's risk assessment 
model continues to evolve, the Coast Guard could benefit from periodic 
external peer review to ensure that the structure and outputs of the 
model are appropriate for its given uses and to identify possible 
areas for improvement. 

Coast Guard Has Used a Risk-Informed Approach to Manage Maritime 
Security Risk, but Challenges Hinder Sector Efforts: 

MSRAM Informs Several National-Level Risk Management Efforts: 

MSRAM is a security risk analysis and risk management tool and the 
Coast Guard intends for it to be used to inform risk management 
decisions at all levels of command.[Footnote 45] As such, in a May 
2011 guidance document, the Coast Guard set expectations for how MSRAM 
should be used at the national and sector levels. At the national 
level, the Coast Guard expects its offices to use MSRAM to support 
strategic plans, policy, and guidance; to integrate MSRAM into 
maritime security programs; and to ensure that sectors have adequate 
personnel ready to perform MSRAM duties, among other goals. 

At the national level, MSRAM assists in the development and 
implementation of several operational and strategic planning efforts, 
which align with the Coast Guard's expectations for how risk 
information should be used.[Footnote 46] One key use of MSRAM data at 
the national level has been to refine the national MCIKR list, which 
the Coast Guard reports has allowed it to focus resources on the 
highest-risk maritime targets. Coast Guard headquarters requires 
sectors to meet certain operational activity standards, such as MCIKR 
visits, patrol frequencies, and vessel escort requirements set under 
the Maritime Security and Response Operations (MSRO) program.[Footnote 
47] By identifying the nation's highest-risk maritime targets, MSRAM 
helps establish the national MCIKR list, which sectors use to complete 
their annually required number of MCIKR visits. According to Coast 
Guard officials, MSRAM has aided in reducing the MCIKR list from 740 
assets to 324 assets and allowed the Coast Guard to further prioritize 
within that more focused list of 324, since MSRAM analysis 
demonstrated that a small number of assets make up the majority of the 
nation's risk.[Footnote 48] 

MSRAM has also been used as a tool to inform resource allocation and 
performance measurement, which is consistent with the Coast Guard's 
goals for MSRAM.[Footnote 49] For instance, risk-informed methods and 
processes or models, such as MSRAM, are used in the Coast Guard's 
annual Standard Operational Planning Process, which establishes a 
standardized process to apportion major assets, such as boats, 
aircraft, and deployable specialized forces. Coast Guard officials 
said that MSRAM data supports the PWCS mission in this process by 
demonstrating how risk is distributed geographically. In addition, 
Coast Guard used MSRAM to support a funding request for boats, 
personnel, and associated support costs to assist with Coast Guard 
efforts to reduce the risk of certain dangerous cargoes by escorting 
ships passing through coastal ports carrying cargoes such as liquefied 
natural gas. MSRAM also supports resource allocation through the Port 
Security Grant Program by informing the risk formula used by DHS to 
allocate grant funding.[Footnote 50] MSRAM data are also used in the 
Coast Guard's model for measuring its performance in the PWCS mission, 
which is discussed in depth later in this report. 

MSRAM has also supported strategic documents and efforts throughout 
DHS. Specifically, the Coast Guard reported that MSRAM data are an 
essential building block for a number of key strategic documents, such 
as the National Maritime Strategic Risk Assessment, the National 
Maritime Terrorism Threat Assessment, and the Combating Marine 
Terrorism Strategic and Performance Plan, among others.[Footnote 51] 
In addition, the Coast Guard uses MSRAM, among other inputs, to 
provide DHS with maritime risk information for the Transportation 
Sector Security Risk Assessment tool.[Footnote 52] DHS also reported 
that the Coast Guard has shared MSRAM-based identification of critical 
assets beyond the transportation system with 13 of the 18 DHS critical 
infrastructure and key resource sectors.[Footnote 53] For example, 
MSRAM has been used to assess the risk of some chemical facilities and 
power plants. 

MSRAM Has Informed Some Local-Level Risk Management Efforts, But Its 
Use Has Been Limited By Several Factors: 

MSRAM has been used to inform a variety of efforts at the sector 
level, such as strategic planning, communication with port 
stakeholders, and operational and tactical decision making, but its 
use for operational and tactical risk management efforts has been 
limited by a lack of staff time, the complexity of the MSRAM tool, and 
competing mission demands, among other factors. The Coast Guard 
expects its 35 sectors, with support from its nine districts, to 
integrate MSRAM data into strategic, operational, and tactical plans, 
operations, and programs as necessary and required, among other 
actions.[Footnote 54] 

Based on results from our interviews with officials from all 35 Coast 
Guard sectors, officials from 26 sectors (74 percent) reported finding 
MSRAM moderately to very useful for informing strategic planning, 
which includes developing portions of local Area Maritime Security 
Plans and planning security exercises.[Footnote 55] For example, 
officials from a Gulf Coast sector reported using MSRAM to find the 
highest-risk areas in which to conduct exercises. Further, lessons 
learned from the exercises are incorporated into strategic plans, 
which officials said leads to planning process improvements and 
overall better plans. However, officials from a southeastern sector 
pointed out that MSRAM is a snapshot view of port risk and therefore 
long-term strategic plans require additional information from many 
sources. 

For communicating risk information to port security stakeholders, such 
as local law enforcement or facility owners, officials from 26 sectors 
(74 percent) said that MSRAM was moderately to very useful. For 
instance, officials from a southeastern sector said that MSRAM is used 
to communicate and justify additional security procedures. Further, 
during annual compliance inspections, MSRAM data are discussed with 
facility security officers and compared to security data that the 
facility security officers have calculated. In addition, officials 
from a Gulf Coast sector reported that MSRAM provides a convenient, 
objective way to communicate risk to port security stakeholders, and 
stakeholders appreciate that risk information from MSRAM is computer 
driven and based on a rigorous process. 

For informing sector operational and tactical decision making, such as 
planning MSRO activities, developing local critical infrastructure 
lists, and planning for special events, officials from 18 sectors (51 
percent) reported that MSRAM moderately or greatly provided them with 
the information needed to make risk-informed decisions regarding port 
security. Regarding planning MSRO activities, one eastern sector 
reported that MSRAM was very helpful for identifying priority targets 
for MSRO patrols and escorts. Regarding developing local critical 
infrastructure lists, officials from an eastern sector said that since 
the sector has no assets on the national MCIKR list, they were able to 
use MSRAM to generate a local list to help determine patrols and other 
security efforts. Regarding special event planning, officials from 16 
sectors (45 percent) told us they used MSRAM to determine where to 
allocate resources for special events, such as the Fourth of July, 
dignitary visits, or political conventions. For example, officials 
from an inland river sector said that they used MSRAM to identify 
possible attack scenarios and to help identify what security resources 
they should request to provide security for a special event. See 
figure 1 for photographs of various Coast Guard security-related 
activities that can be informed by MSRAM. In addition to using MSRAM 
to inform maritime security decisions, officials from almost every 
sector noted that they also assess and manage risk using other tools 
or methods, such as the High Interest Vessel matrix, outreach to port 
partners, working relationships with Area Maritime Security 
Committees, or professional judgment.[Footnote 56] 

Figure 1: MSRAM Can Be Used to Inform a Variety of Coast Guard 
Activities and Operations, Including Escorting Ferries, Naval Vessels, 
or Cruise Ships and Performing Waterborne or Aerial Patrols of 
Critical Infrastructure or National Symbols: 

[Refer to PDF for image: 6 photographs] 

Source: GAO. 

[End of figure] 

Although officials from most sectors found that MSRAM provided useful 
risk information for sector-level decision making, officials from 32 
sectors (91 percent) reported that their overall use of MSRAM data in 
managing risk was hindered by a lack of staff time for data analysis, 
the complexity of the MSRAM tool, or competing mission demands, among 
other things. These challenges are discussed below. 

Limited staff time for analyzing and using MSRAM. Officials from 21 
sectors (60 percent) told us that limited staff time posed a challenge 
to incorporating MSRAM into strategic, operational, and tactical 
planning efforts. For example, officials from a northeastern sector 
said that a lack of available staff time was one of the most 
significant limitations to utilizing MSRAM. These officials stated 
that they would like to have dedicated MSRAM personnel to develop the 
tool and make it useful on a daily basis. They added that even though 
MSRAM had many capabilities, they were unable to use it to its full 
capability because their port security specialist--the primary user of 
MSRAM--was busy with other programs, such as the Port Security Grant 
Program. Each of the port security specialists from the three 
districts we interviewed--which encompass 15 sectors over the West 
Coast, East Coast, Gulf Coast, and Mississippi River area--echoed the 
challenges with the level of sector resources for MSRAM.[Footnote 57] 
For example, one district official stated that although Coast Guard 
headquarters has dedicated MSRAM staff, there are no full-time MSRAM 
analysts at the sector level. He added that each sector would need a 
dedicated person for MSRAM and risk analysis to bring MSRAM analysis 
into operational and tactical decision making. 

Complexity of the MSRAM tool. Officials from 14 sectors (40 percent) 
reported that MSRAM use has been limited because data outputs require 
a substantial degree of analysis to use in decision making, or because 
the MSRAM tool itself is not easy to use. Some of the challenges 
raised by sectors that contribute to the complexity of the tool and 
interpreting its outputs included keeping abreast of yearly changes to 
the MSRAM tool and bridging knowledge gaps that occur when staff 
familiar with MSRAM rotate or leave the sector. In its MSRAM core 
document, the Coast Guard recognized that the frequent rotation of 
active duty personnel presents a risk to both the consistency of the 
MSRAM risk scoring efforts and the application of risk results. 

Competing mission demands and resource constraints. Officials from 14 
sectors (40 percent) reported that competing mission demands or 
resource constraints limited the use of MSRAM. Specifically, officials 
from 11 sectors reported that MSRAM's usefulness was limited by the 
fact that it only considers risk in the PWCS mission, which is 1 of 
the Coast Guard's 11 statutorily required missions. For example, a 
Great Lakes sector told us that while MSRAM identifies the risks in 
the sector, the sector is limited in its ability to move assets to 
address those security risks because the assets are also fulfilling 
other Coast Guard mission requirements, such as search and rescue. 
Additionally, officials from 6 sectors said that limited resources, 
such as boats or personnel, constrained their sectors' ability to 
address the risks identified by MSRAM. For example, officials from 2 
inland river sectors said that MSRAM identifies their security risks 
and demonstrates where they should patrol and plan for special events, 
but that they do not have the resources to carry out the plans. 
Further, officials from 1 of the inland river sectors added that their 
response boats are often busy escorting the Army Corps of Engineers or 
engaged in flood relief efforts. This leaves the work of security 
patrols to the local harbor patrol, which the officials said does not 
have the same capabilities, in terms of boats and weapons, as the 
Coast Guard. 

Other challenges. Sector officials also identified other challenges 
with using MSRAM for informing decision making. Specifically, 
officials from 16 sectors (45 percent) said that MSRAM would be more 
useful if it was linked to other Coast Guard data systems, such as the 
Coast Guard's inspections database, or if MSRAM was integrated into 
the sector command center. For example, officials from an east coast 
sector told us that they would like to see MSRAM linked to other 
databases in the sector command center, such as the Coast Guard's 
vessel tracking system. Similarly, officials from a west coast sector 
said that integrating MSRAM into the Coast Guard's inspections 
database would keep MSRAM continually updated and reflective of 
inspection results. Further, the command center has to consider other 
mission response needs, such as for pollution incidents or search and 
rescue, among others, and if MSRAM was integrated into the sector 
command center it could be used more in day-to-day operations. In 
addition, officials from 5 sectors noted that MSRAM does not capture 
dynamic risk, which limits its ability to inform daily decisions at 
the sector level.[Footnote 58] For instance, officials from a Gulf 
Coast sector said that they did not use MSRAM on a daily basis to 
allocate resources because daily fluctuations in vessel and barge risk 
are their greatest concern and this risk is not currently captured in 
MSRAM. The sectors that raised these issues believed that linking 
MSRAM into other data systems, integrating MSRAM into the command 
center, and having MSRAM account for dynamic risks could contribute to 
making its data more accurate, robust, and useful for decision making. 

Coast Guard Has Taken Steps To Address Challenges, But Could Increase 
MSRAM'S Use By Expanding Training Opportunities: 

Coast Guard headquarters officials told us that they were aware of the 
challenges field-level MSRAM users were facing and have taken some 
steps to address them, but providing additional training could help 
integrate MSRAM throughout sector decision making. The Coast Guard's 
current actions to address MSRAM user challenges include assessing the 
feasibility of adding additional risk analyst staff, increasing the 
data's usability, developing decision-supporting modules, and 
providing training. These actions are described below. 

Examining the feasibility of dedicated risk analysts. Presently, there 
is no dedicated risk analyst or MSRAM analyst position at the sector 
level, but headquarters officials told us in June 2011 that they are 
examining the feasibility of assigning additional port security 
specialists to the field and submitted a resource proposal for the 
additional staff. According to a senior Coast Guard budget official, 
given competing priorities and a constrained resource environment, it 
is unclear when or if this resource proposal will be funded. 

Deploying MSRAM to sector command centers. To help make MSRAM more 
dynamic and increase its usability, the Coast Guard is piloting an 
Enterprise Geographic Information System (EGIS) display for sector 
command centers, which layers facility and vessel locations onto a 
satellite-based map and visually displays changing risk as vessels 
move into and out of ports. Officials from 7 sectors that participated 
in or were familiar with the initial EGIS test group reported that the 
functionality was very useful and had the potential to substantially 
increase MSRAM's use for sector risk management efforts. In addition, 
headquarters officials told us in June 2011 that efforts were under 
way to integrate MSRAM into the Coast Guard's inspections database, 
which would allow MSRAM to be continually updated and reflective of 
year-round facility and vessel inspection results. 

Developing risk management modules. To assist with incorporating risk 
assessment information into decision making, in the fall of 2008, the 
Coast Guard began developing risk management modules within MSRAM that 
are able to provide specific types of analyses, such as comparing 
alternative security strategies. We asked officials from all 35 
sectors their views on four modules--the Alternatives Evaluation 
Module, the Simplified Reporting Interface, the Daily Risk Profile, 
and the Risk Management Module.[Footnote 59] Sectors had mixed views 
on the utility of these modules. Specifically, officials from 14 
sectors (40 percent) found the Alternatives Evaluation module very 
useful and cited such uses as evaluating Port Security Grant Program 
proposals and planning security for special events,[Footnote 60] and 
officials from 15 sectors (42 percent) found the Simplified Reporting 
Interface very useful for communicating risk information to port 
partners.[Footnote 61] However, with respect to the other two modules--
the Daily Risk Profile and Risk Management Module--officials from 2 
sectors (5 percent) found the Daily Risk Module very useful and 
officials from 3 sectors (8 percent) found the Risk Management Module 
very useful. For both modules, officials from 18 sectors (51 percent) 
reported that either they had not seen them or they were aware of the 
modules but did not have the time or training, among other reasons, to 
use them. Many of the modules are new and headquarters and some sector 
officials reported that they expected the modules would be more useful 
in the future as sectors gained familiarity with them through 
additional exposure and the annual MSRAM training. 

Providing training. While the Coast Guard offers annual MSRAM 
training, officials from 25 sectors (71 percent) identified areas of 
the training for improvement, which the Coast Guard could do more to 
address.[Footnote 62] Specifically, officials from these sectors said 
that increasing the number of people who take MSRAM training, 
providing MSRAM training to command-level staff or senior management, 
and offering training on how to conduct risk analysis to inform 
decision making, among other things, would help integrate MSRAM 
throughout sector decision-making processes. Since MSRAM is a 
collateral duty, MSRAM training is not part of any Coast Guard 
personnel's required training curriculum.[Footnote 63] However, Coast 
Guard guidance from May 2011 states that area, district, and sector 
commanders are responsible for ensuring that adequate numbers of 
appropriate personnel are trained in MSRAM. Only one sector did not, 
at the time of our interview, have at least one staff person trained 
in MSRAM.[Footnote 64] Officials from a Gulf Coast sector said that 
the training provided on the MSRAM tool itself is good, but the 
training does not teach the skills needed to make decisions in the 
field. Officials from a Great Lakes sector suggested that the Coast 
Guard develop an advanced course on how to use MSRAM to inform 
operational decisions. Officials from a southeastern sector added that 
the Coast Guard provides guidance on how to assess risks using MSRAM, 
but needs to provide more training on how to communicate MSRAM results 
and how those results can be used. In addition, a sector commanding 
officer who participated in one of our interviews told us that he was 
provided minimal training on MSRAM and wanted to understand more about 
how it can be used to support command-level decisions. 

MSRAM has the capability of informing operational, tactical, and 
resource allocation decisions at all levels of a sector, but the Coast 
Guard has generally provided MSRAM training to a limited number of 
sector staff with specific MSRAM risk assessment responsibilities, 
such as port security specialists, rather than sector staff who may 
have command or management responsibilities where MSRAM may apply. 
Coast Guard headquarters officials said that this was because of 
limited resources to provide training for numerous sector personnel 
and variations in how MSRAM responsibilities are managed at different 
sectors. Standards for Internal Control in the Federal Government 
states that effective management of an organization's workforce is 
essential to achieving results. Further, only when the right personnel 
for the job are on board and are provided the right training and 
tools, among other things, is operational success possible. To this 
end, management should ensure that training is aimed at developing and 
retaining employee skill levels to meet changing organizational needs. 
[Footnote 65] Coast Guard headquarters officials agree that providing 
MSRAM training to additional sector staff, particularly those with 
command and management responsibilities, would be valuable. Such 
training on how MSRAM can be used at all levels of command for risk-
informed decision making--including how MSRAM can assist with the 
selection of different types of security measures to address areas of 
risk and the evaluation of their impacts--could further the Coast 
Guard's efforts to implement its risk management framework and meet 
its goal to institutionalize MSRAM as the risk management tool for 
maritime security. 

Coast Guard Measures Risk Reduction but Has Faced Challenges Using 
This Measure to Inform Decisions: 

Coast Guard Developed A Measure To Report Performance In Reducing Risk: 

The Coast Guard developed a performance measure and supporting model 
to measure and report its overall performance in reducing maritime 
security risk. This measure identifies the percentage reduction of 
maritime security risk, subject to Coast Guard influence, resulting 
from various Coast Guard actions.[Footnote 66] The Coast Guard 
considers this performance measure its key outcome measure for its 
PWCS mission.[Footnote 67] According to DHS's Risk Management 
Fundamentals and the NIPP, it is crucial that a process of performance 
measurement be established to evaluate whether actions taken 
ultimately achieve the intended performance objective, such as 
reducing risk. This is important not only in evaluating program 
performance but also in holding the organization accountable for 
progress. We have also previously reported on the importance of 
developing outcome-based performance goals and measures as part of 
results management efforts.[Footnote 68] From fiscal years 2006 to 
2010, the Coast Guard annually reported reducing from 15 to 31 percent 
of the maritime risk it is responsible for, in each year either 
meeting or exceeding its target. For fiscal years 2011 and 2012, the 
Coast Guard's planned performance targets are to reduce more than 44 
percent of the maritime security risk for which it is responsible. 
[Footnote 69] 

To measure how its actions have reduced risk, the Coast Guard 
developed a model that uses a two-step approach. The first step is to 
estimate the total amount of terrorism risk that exists in the 
maritime domain, in the absence of any Coast Guard activities. This is 
referred to as raw risk, and this information comes primarily from 
MSRAM.[Footnote 70] The second step relies on an elicitation process 
whereby Coast Guard subject matter experts estimate how various 
security activities and operations, maritime domain awareness 
programs, and regulatory structures--referred to by the Coast Guard as 
regimes--that the Coast Guard has implemented have reduced risk to 
U.S. ports and waterways. This step involves Coast Guard subject 
matter experts assessing the probability of these Coast Guard efforts 
failing to prevent a successful terrorist attack for 16 potential 
maritime terrorist attack scenarios.[Footnote 71] 

Eliciting the opinions of subject matter experts is a method that can 
be used to estimate terrorism risk, particularly when the historical 
record is either nonexistent or is not appropriate for collecting data 
on a specific scenario, according to DHS's Risk Management 
Fundamentals.[Footnote 72] The Coast Guard relies on subject matter 
experts to estimate performance in mitigating risk because, unlike 
risk analyses for other Coast Guard missions, such as search and 
rescue, there is not a rich historical data set of maritime terrorism 
incidents that the Coast Guard can use to measure its actual 
performance. In other words, in the absence of an actual domestic 
maritime terrorism event, the Coast Guard uses internal subject matter 
experts to estimate risk reduction as a proxy measure of performance--
an attempt to measure performance against a terrorism incident that 
did not occur.[Footnote 73] 

Coast Guard Faces Challenges Using The Risk Reduction Performance 
Measure To Inform Decision Making: 

The Coast Guard's efforts to develop an outcome measure to quantify 
the impact its actions have had on risk is a positive step. However, 
the use of the measure has been limited, and even with recent 
improvements, the Coast Guard faces challenges using this measure to 
inform decision making. Performance goals and measures are intended to 
provide Congress and agency management with information to 
systematically assess a program's strengths, weaknesses, and 
performance. Thus, measures should provide information for management 
decision making. Coast Guard officials explained that the primary 
purpose of the risk reduction measure has been for external 
performance reporting, and to a more limited extent for informing 
strategic decision making and for conducting internal analysis of 
performance to identify areas for improvement. Specifically, officials 
said the measure has been used to compare risk across maritime 
terrorism scenarios and compare those results to other studies and 
analysis on maritime terrorism scenarios, which provided information 
on whether PWCS activities were appropriately balanced to address 
those risks. However, Coast Guard officials stated that over time, 
internal and external reviews identified limitations in the risk 
reduction measure, such as not allowing for comparisons of performance 
across sectors.[Footnote 74] 

Recognizing these limitations, in 2010, the Coast Guard made 
improvements to the risk reduction model intended to enhance its 
utility for management decision making and to provide a more accurate 
measure of risk reduction. For example, the updated model includes 
information on the locations of Coast Guard assets and potential 
targets, which can be used to calculate the probability that Coast 
Guard assets will be able to intercept attacks. The Coast Guard also 
improved the elicitation techniques by which subject matter experts 
provided their estimates of Coast Guard risk reduction performance, 
and expanded the size and diversity of the subject matter experts 
involved in the elicitation process.[Footnote 75] According to Coast 
Guard officials, these improvements have made the measure and 
supporting model more useful for informing strategic decisions by 
allowing, for example, the ability to calculate risk reduction at the 
sector, district, area, and national levels and the risk reduction 
value of each element of the Coast Guard's strategy. In other words, 
the updated model is able to show the risk reduction value of Coast 
Guard operational assets, such as small boats or helicopters, compared 
with regime activities, such as regulation enforcement. This 
information can help inform resource allocation decisions because it 
could identify which actions provide the greatest risk-reduction, 
according to these officials. The Coast Guard plans to use the updated 
model to measure its performance in reducing risk for the 2011 fiscal 
year. 

Since the Coast Guard has not yet used the new measure or supporting 
model for management analysis and decision making, it is too soon to 
determine how useful the information will ultimately be for Coast 
Guard decision makers or external stakeholders; nevertheless, the 
Coast Guard may continue to face challenges using this measure to 
inform decision making. For example, given the inherent uncertainties 
in estimating risk reduction, it is unclear if a measure of risk 
reduction would provide meaningful performance information for 
tracking progress against goals and performance over time. According 
to our performance measurement criteria, to be able to assess progress 
toward the achievement of performance goals, the measures used must be 
reliable and valid.[Footnote 76] Reliability refers to the precision 
with which performance is measured, while validity is the extent to 
which the measure adequately represents actual performance. Therefore, 
the usefulness of agency performance information depends to a large 
degree on the reliability of performance data. We have also reported 
that decision makers must have assurance that the program data being 
used to measure performance are sufficiently reliable and valid if the 
data are to inform decision making.[Footnote 77] Although the Coast 
Guard has taken steps to improve the quality of the supporting model 
to provide a more accurate measure, estimating risk reduction is 
inherently uncertain and this measure is based on largely subjective 
judgments of Coast Guard personnel, and therefore the risk reduction 
results reported by the Coast Guard are not based on measurable or 
observable activities.[Footnote 78] As a result, it is difficult to 
independently verify or assess the validity or appropriateness of the 
judgments or to determine if this is an accurate measure of Coast 
Guard performance in the PWCS mission. However, Coast Guard officials 
told us that they believe these reported results provide a useful 
proxy measure of Coast Guard performance, and noted that this is one 
of several metrics the Coast Guard uses to assess performance in the 
PWCS mission. 

According to DHS's Risk Management Fundamentals, it is also important 
to be transparent about assumptions and key sources of uncertainty, so 
that decision makers are informed of the limitations of the risk 
information provided by the model. In its 2009 review of the risk 
reduction model, CREATE at the University of Southern California 
stated that it seemed likely that the model ignored important 
uncertainties and implied incorrectly high precision of risk 
estimates.[Footnote 79] Furthermore, OMB's Updated Principles for Risk 
Analysis notes that because of the inherent uncertainties associated 
with estimates of risk, presentation of a single risk estimate may be 
misleading and provide a false sense of precision. OMB suggests that 
when a quantitative characterization of risk is provided, a range of 
plausible risk estimates should also be provided.[Footnote 80] From 
fiscal years 2006 to 2010, the Coast Guard reported the risk reduction 
measure as a specific risk reduction number rather than as a range of 
plausible risk reduction estimates. The Coast Guard official 
responsible for this measure told us this was because the previous 
risk reduction model was not capable of producing a range of plausible 
risk reduction estimates. The official noted that while the new risk 
reduction model--which will be used to report results for fiscal year 
2011--is capable of producing a range of estimated risk reduction, the 
Coast Guard will continue to report the risk reduction measure as a 
single number because the DHS data system for performance reporting 
does not accept ranges--only numerical values. However, the official 
added that there is value in reporting a range of risk reduction and 
officials are considering a transition to a range of estimated 
reduction for the PWCS mission in future years. One alternative could 
be to report the percentage of risk reduced as a single number, but 
having an explanatory note indicating the range of plausible risk 
reduction estimates. Using a risk reduction measure that more 
accurately reflects performance effectiveness can give Coast Guard 
leaders and Congress a better sense of progress toward goals, which 
can support efforts to identify areas for improvement. 

DHS officials have also raised some questions about the risk reduction 
measure. Recently, DHS determined that the Coast Guard's risk 
reduction measure was not appropriate for inclusion as a DHS strategic 
performance measure and has designated it as a management measure. 
According to DHS, a strategic measure is designed to communicate 
achievement of strategic goals and objectives and be readily 
understandable to the public, and a management measure is designed to 
gauge program results and tie to resource requests and be used to 
support achievement of strategic goals. According to a senior DHS 
official, in 2010, DHS leadership reviewed all existing department 
measures and made decisions about which measures they believed were 
clearly tied to the DHS Quadrennial Homeland Security Review missions 
and were easily understandable by the public.[Footnote 81] This 
official noted that based on this review, DHS leadership did not feel 
the risk reduction measure and its methodology would be easily 
understandable by the public and therefore did not designate the 
measure as a strategic measure. As a result, the risk reduction 
measure will not be included in DHS's annual performance plan, 
formally published with the Annual Performance Report, because this 
report only includes the smaller set of strategic measures.[Footnote 
82] However, this official noted that the risk reduction measure is 
important as one piece of information to manage risk and is considered 
to be part of the full suite of DHS performance measures, and will 
continue to be published in the Coast Guard's strategic context that 
is submitted with DHS's Annual Performance Report.[Footnote 83] 

Conclusions: 

The Coast Guard has invested substantial effort incorporating risk 
management principles into its security priorities and investments, 
and continues to proactively strengthen its assessment, management, 
and evaluation practices. As a result, the Coast Guard's risk 
assessments and risk model are generally sound and in alignment with 
DHS standards. However, there are some additional actions that the 
Coast Guard could take to further its risk management approach by 
facilitating a wider use of risk information and making the results 
more valuable to the users. For example, since risk management is a 
tool for informing policymakers' decisions about assessing risks, 
allocating resources, and taking actions under conditions of 
uncertainty, the Coast Guard could better document and communicate the 
uncertainty or confidence levels of its risk assessment results, 
including any implications that the uncertainty may have for decision 
makers. This added information would allow Coast Guard decision makers 
to prioritize strategies, tactics, and long-term investments with 
greater insight about the range of likely results and associated trade-
offs with each decision. Additional information would also allow 
external reviewers of the risk model to reach the most appropriate 
conclusions or provide the most useful improvement recommendations 
through periodic reviews. The Coast Guard could also enhance the risk-
informed prioritization of its field-level strategies, operations, and 
tactics by ensuring that risk management training is expanded to 
multiple levels of Coast Guard decision makers at the sector level, 
including command-level personnel. Expanding training on how MSRAM 
could be used at all levels of command for risk-informed decision 
making--including how MSRAM can assist with the selection of different 
types of security measures and the evaluation of their impacts--would 
further the Coast Guard's efforts to implement its risk management 
framework and meet its goal of institutionalizing MSRAM as the risk 
management tool for maritime security. Finally, accurately 
representing performance results is important and the Coast Guard 
could more accurately convey its risk reduction performance measure by 
reporting risk reduction results as a range rather than a point 
estimate. Presenting risk reduction as a single number without a 
corresponding range of uncertainty could hamper Coast Guard efforts to 
identify areas for improvement. Taking these steps would make the 
Coast Guard's risk management approach even stronger. 

Recommendations for Executive Action: 

To help the Coast Guard strengthen MSRAM and better align it with NIPP 
risk management guidance, as well as facilitate the increased use of 
MSRAM across the agency, we recommend that the Commandant of the Coast 
Guard take the following three actions: 

(1) Provide more thorough documentation related to key assumptions and 
sources of uncertainty within MSRAM and inform users of any 
implications for interpreting the results from the model. 

(2) Make MSRAM available to appropriate parties for additional 
external peer review. 

(3) Provide additional training for sector command staff and others 
involved in sector management and operations on how MSRAM can be used 
as a risk management tool to inform sector-level decision making. 

To improve the accuracy of the risk reduction measure for internal and 
external decision-making, we recommend that the Commandant of the 
Coast Guard take action to report the results of the risk reduction 
measure as a range rather than a point estimate. 

Agency Comments and Our Evaluation: 

We provided a draft of this report to DHS and the Coast Guard on 
October 17, 2011, for review and comment. DHS provided written 
comments, which are reprinted in appendix II. DHS and the Coast Guard 
concurred with the findings and recommendations in the report, and 
stated that the Coast Guard is taking actions to implement our 
recommendations. 

The Coast Guard concurred with our first recommendation that it 
provide more thorough documentation related to key assumptions and 
sources of uncertainty within MSRAM. Specifically, the Coast Guard 
stated that the documentation of uncertainty is part of the ongoing 
MSRAM VV&A process, and that the Coast Guard will continue to work 
with the DHS Office of Risk Management and Analysis in developing a 
feasible and deployable model that will benefit field-level security 
operations. These actions should improve the Coast Guard's ability to 
document and inform MSRAM users of any implications for interpreting 
results from the model, thereby addressing the intent of our 
recommendation. 

Regarding the second recommendation that the Coast Guard make MSRAM 
available to appropriate parties for additional external peer review, 
the Coast Guard concurred. The Coast Guard stated that external peer 
review is part of the ongoing MSRAM VV&A process, and that additional 
external peer review will be part of an independent verification and 
validation of MSRAM expected to be completed in the fall of 2012. Such 
actions should address the intent of the recommendation. 

Regarding the third recommendation that the Coast Guard provide 
additional training for sector command staff and others involved in 
sector management on how MSRAM can be used as a risk management tool, 
the Coast Guard concurred. Specifically, the Coast Guard stated that 
MSRAM is part of the Coast Guard's contingency planning course, and 
the Coast Guard will explore other opportunities to provide risk 
training to sector command staff, including online and webinar 
training opportunities. Such actions, once implemented, should address 
the intent of the recommendation. 

Finally, the Coast Guard also concurred with the fourth recommendation 
to take action to report the results of the risk reduction measure as 
a range rather than a point estimate. The Coast Guard stated that it 
is currently limited by the DHS data reporting system with regard to 
the format of presenting performance targets and results, but noted 
that it is currently working with DHS to determine options for 
reporting risk as a range. Such action, when fully implemented, should 
address the intent of the recommendation. 

DHS and the Coast Guard also provided us with technical comments, 
which we incorporated as appropriate. 

As agreed with your offices, unless you publicly announce the contents 
of this report earlier, we plan no further distribution until 30 days 
from the report date. At that time, we will send copies to the 
Secretary of Homeland Security, appropriate congressional committees, 
and other interested parties. In addition, the report will be 
available at no charge on the GAO website at [hyperlink, 
http://www.gao.gov]. 

If you or your staff have any further questions about this report, 
please contact me at (202) 512-9610 or caldwells@gao.gov. Contact 
points for our Offices of Congressional Relations and Public Affairs 
may be found on the last page of this report. Key contributors are 
listed in appendix III. 

Signed by: 

Stephen L. Caldwell: 
Director, Homeland Security and Justice Issues: 

[End of section] 

Appendix I: Risk Management Framework: 

To provide guidance to agency decision makers, we developed a risk 
management framework which is intended to be a starting point for 
applying risk-informed principles.[Footnote 84] Our risk management 
framework, shown in figure 2, entails a continuous process of managing 
risk through a series of actions, including setting strategic goals 
and objectives, assessing risk, evaluating alternatives, selecting 
initiatives to undertake, and implementing and monitoring those 
initiatives. 

Figure 2: GAO's Risk Management Framework: 

[Refer to PDF for image: illustration] 

Interlocking circular segments: 

Strategic Goals, Objectives, and Constraints: 
Risk Assessment: 
Alternatives Evaluation: 
Management Selection: 
Implementation and Monitoring: 

Source: GAO. 

[End of figure] 

Setting strategic goals, objectives, and constraints is a key first 
step in applying risk management principles and helps to ensure that 
management decisions are focused on achieving a purpose. Risk 
assessment, an important element of a risk-informed approach, helps 
decision makers identify and evaluate potential risks so that 
countermeasures can be designed and implemented to prevent or mitigate 
the effects of the risks. Risk assessment is a qualitative 
determination, quantitative determination, or both of the likelihood 
of an adverse event occurring and the severity, or impact, of its 
consequences. Risk assessment in a homeland security application 
involves assessing three key components--threat, vulnerability, and 
consequence. A threat assessment is the identification and evaluation 
of adverse events that can harm or damage an asset. A vulnerability 
assessment identifies weaknesses in physical structures, personal 
protection systems, processes, or other areas that may be exploited. A 
consequence assessment is the process of identifying or evaluating the 
potential or actual effects of an event, incident, or occurrence. 
Information from these three assessments contributes to an overall 
risk assessment that characterizes risks, which can provide input for 
evaluating alternatives and prioritizing security initiatives. The 
risk assessment element in the overall risk management cycle informs 
each of the remaining steps of the cycle. Alternatives evaluation 
addresses the evaluation of risk reduction methods by consideration of 
countermeasures or countermeasure systems and the costs and benefits 
associated with them. Management selection addresses such issues as 
determining where resources and investments will be made, the sources 
and types of resources needed, and where those resources would be 
targeted. The next phase in the framework involves the implementation 
of the selected countermeasures. Following implementation, monitoring 
is essential to help ensure that the entire risk management process 
remains current and relevant and reflects changes in the effectiveness 
of the alternative actions and the risk environment in which it 
operates. Program evaluation is an important tool for assessing the 
efficiency and effectiveness of the program. As part of monitoring, 
consultation with external subject area experts can provide a current 
perspective and an independent review in the formulation and 
evaluation of the program. 

The National Infrastructure Protection Plan (NIPP), originally issued 
by the Department of Homeland Security (DHS) in 2006 and updated in 
2009, includes a risk analysis and management framework, which, for 
the most part, mirrors our risk management framework. This framework 
includes six steps--set goals and objectives; identify assets, 
systems, and networks; assess risks; prioritize; implement programs; 
and measure effectiveness. The NIPP is DHS's base plan that guides how 
DHS and other relevant stakeholders should use risk management 
principles to prioritize protection activities. In 2009, DHS updated 
the NIPP to, among other things, increase its emphasis on risk 
management, including an expanded discussion of risk management 
methodologies and discussion of a common risk assessment approach that 
provided core criteria for these analyses. Beyond the NIPP, DHS has 
issued additional risk management guidance and directives. For 
example, in January 2009 DHS published its Integrated Risk Management 
Framework, which, among other things, calls for DHS to use risk 
assessments to inform decision making. In April 2011, DHS issued its 
Risk Management Fundamentals, which establishes specific doctrine and 
guidance for risk management across DHS. 

[End of section] 

Appendix II: Comments from the Department of Homeland Security: 

U.S. Department of Homeland Security: 
Washington, DC 20528: 

November 7, 2011: 

Steven Caldwell: 
Director, Homeland Security and Justice: 
U.S. Government Accountability Office: 
441 G Street, NW: 
Washington, DC 20548: 

Re: Draft Report GAO-12-14, "Coast Guard: Security Risk Model Meets 
DHS Criteria, but More Training Could Enhance Its Use for Managing 
Programs and Operations" 

Dear Mr. Caldwell: 

Thank you for the opportunity to review and comment on this draft 
report. The U.S. Department of Homeland Security (DHS) appreciates the 
U.S. Government Accountability Office's (GAO's) work in planning and 
conducting its review and issuing this report. 

As the lead federal agency for maritime security, including the 
protection of U.S. ports, coasts, and inland waterways, the U.S. Coast 
Guard's mission involves protecting the maritime domain and marine 
transportation system, including preventing terrorists attacks, and 
responding to and recovering from attacks. To assist with this 
mission, since 2005, the Coast Guard has been using its Maritime 
Security Risk Analysis Model (MSRAM) as its primary approach for 
assessing and managing security risks, which has provided the Coast 
Guard with a standardized way of assessing risk of the maritime 
infrastructure. 

The Department is pleased to note GAO's positive acknowledgment of the 
substantial efforts the Coast Guard had made incorporating risk 
management principles into its security priorities and investments. 
The draft report also recognizes that the Coast Guard continues to 
proactively strengthen its assessment, management and evaluation 
practices and, as a result, MSRAM generally meets DHS criteria for 
being complete, reproducible, documented, and defensible. The
Coast Guard is taking action and developing long-term strategies to 
further improve MSRAM methodology and risk assessment process. It is 
important to note that related challenges are not unique to MSRAM or 
the Coast Guard, but faced by many others in the Homeland Security 
risk assessment community. 

The draft report contained four recommendations with which the 
Department concurs. Specifically, GAO recommended that the Commandant 
of the Coast Guard: 

Recommendation 1: Provide more thorough documentation related to key 
assumptions and sources of uncertainty within MSRAM and inform users 
of any implications for interpreting the results from the model. 

Response: Concur. Documentation of uncertainty is part of the ongoing 
MSRAM Verification, Validation & Accreditation (VV&A) process, in 
coordination with DHS, The Coast Guard is analyzing how best to 
incorporate uncertainty estimates as part of the MSRAM risk results. 
As many of the assumptions and sources of uncertainty stem from a lack 
of existing functional, rather than theoretical, security risk models, 
the Coast Guard will continue to work with the DHS National Protection 
and Programs Directorate (NPPD) Office of Risk Management and Analysis
(RMA) personnel in developing a feasible and deployable model that 
will benefit field-level security operations. 

Recommendation 2: Make MSRAM available to appropriate parties for 
additional external peer review. 

Response: Concur. External peer review is part of the ongoing MSRAM 
VV&A process, in coordination with NPPD RMA. The Coast Guard has 
already conducted an independent verification and validation (IV&V) of 
MSRAM, completed in March 2010, and is in the process of conducting a 
second IV&V as part of the MSRAM VV&A accreditation process. External 
peer reviews were included in the first IV&V and will be part of the 
second IV&V, which is expected to be completed in the fall of 2012. 

Recommendation 3: Provide additional training for sector command staff 
and others involved in sector management and operations on how MSRAM 
can be used as a risk management tool to inform sector-level decision 
making. 

Response: Concur. MSRAM is currently part of the "Risk-Based Decision 
Making" lesson taught within the Coast Guard's Contingency Planning 
Course. The lesson objectives are to identify how risk-based decision 
making will impact preparedness planning, and demonstrate the uses of 
MSRAM to develop preparedness plans. The Coast Guard will also 
continue to explore other opportunities to provide risk training to 
Sector command staff, including online and Webinar training 
opportunities. 

Recommendation 4: Take action to report the results of the risk 
reduction measure as a range rather than a point estimate. 

Response: Concur. The Coast Guard is currently limited by the DHS data 
reporting system with regard to the format (i.e. performance targets 
and results). The Coast Guard is currently working with DHS 
OCFO/Program Analysis & Evaluation to determine options for reporting 
risk as a range vice a single number. 

Again, thank you for the opportunity to review and comment on this 
draft report. Technical and sensitive comments were provided under 
separate cover. We look forward to working with you on future Homeland 
Security issues. 

Sincerely, 

Signed by: 

Jim H. Crumpacker: 
Director: 
Departmental GAO-OIG Liaison Office: 

[End of section] 

Appendix III: GAO Contact and Staff Acknowledgments: 

GAO Contact: 

Stephen L. Caldwell, (202) 512-9610 or caldwells@gao.gov: 

Staff Acknowledgments: 

In addition to the contact named above, Dawn Hoff, Assistant Director 
and Adam Hoffman, Analyst-in-Charge, managed this assignment. Chuck 
Bausell, Charlotte Gamble, and Grant Sutton made significant 
contributions to this report. Colleen McEnearney provided assistance 
with interviews and data analysis. Michele Fejfar assisted with 
design, methodology, and data analysis. Jessica Orr provided 
assistance with report development, and Geoff Hamilton provided legal 
assistance. 

[End of section] 

Related GAO Products: 

Port Security Grant Program: Risk Model, Grant Management, and 
Effectiveness Measures Could Be Strengthened. [hyperlink, 
http://www.gao.gov/products/GAO-12-47]. Washington, D.C.: November 17, 
2011. 

Maritime Security: Progress Made but Further Actions Needed to Secure 
the Maritime Energy Supply. [hyperlink, 
http://www.gao.gov/products/GAO-11-883T]. Washington, D.C.: August 24, 
2011. 

Maritime Security: DHS Progress and Challenges in Key Areas of Port 
Security. [hyperlink, http://www.gao.gov/products/GAO-10-940T]. 
Washington, D.C.: July 21, 2010. 

Maritime Security: Varied Actions Taken to Enhance Cruise Ship 
Security, but Some Concerns Remain. [hyperlink, 
http://www.gao.gov/products/GAO-10-400]. Washington, D.C.: April 9, 
2010. 

Critical Infrastructure Protection: Update to National Infrastructure 
Protection Plan Includes Increased Emphasis on Risk Management and 
Resilience. [hyperlink, http://www.gao.gov/products/GAO-10-296]. 
Washington, D.C.: March 5, 2010. 

Transportation Security: Comprehensive Risk Assessments and Stronger 
Internal Controls Needed to Help Inform TSA Resource Allocation. 
[hyperlink, http://www.gao.gov/products/GAO-09-492]. Washington, D.C.: 
March 27, 2009. 

Department of Homeland Security: Progress Report on Implementation of 
Mission and Management Functions. [hyperlink, 
http://www.gao.gov/products/GAO-07-454]. Washington, D.C.: August 17, 
2007. 

Risk Management: Further Refinements Needed to Assess Risks and 
Prioritize Protective Measures at Ports and Other Critical 
Infrastructure. [hyperlink, http://www.gao.gov/products/GAO-06-91]. 
Washington, D.C.: December 15, 2005. 

Managing for Results: Enhancing Agency Use of Performance Information 
for Management Decision Making. [hyperlink, 
http://www.gao.gov/products/GAO-05-927]. Washington, D.C.: September 
9, 2005. 

Homeland Security: Summary of Challenges Faced in Targeting Oceangoing 
Cargo Containers for Inspection. [hyperlink, 
http://www.gao.gov/products/GAO-04-557T]. Washington D.C.: March 31, 
2004. 

Managing for Results: Challenges Agencies Face in Producing Credible 
Performance Information. [hyperlink, 
http://www.gao.gov/products/GAO/GGD-00-52]. Washington, D.C.: February 
4, 2000. 

The Results Act: An Evaluator's Guide to Assessing Agency Annual 
Performance Plans. [hyperlink, 
http://www.gao.gov/products/GAO/GGD-10.1.20]. Washington, D.C.: April 
1998. 

[End of section] 

Footnotes: 

[1] The Coast Guard's 11 statutory missions are (1) PWCS, (2) migrant 
interdiction, (3) defense readiness, (4) drug interdiction, (5) other 
law enforcement, (6) search and rescue, (7) living marine resources, 
(8) Aids to Navigation, (9) ice operations, (10) marine environmental 
protection, and (11) marine safety. 6 U.S.C. § 468. Organizationally, 
the Coast Guard is divided into headquarters and several field 
commands including; the Atlantic and Pacific Areas; 9 districts; and 
35 sectors, which are operational units that carry out the full range 
of Coast Guard missions. There are 35 geographically-based Coast Guard 
sectors in the continental United States, Alaska, Hawaii, Puerto Rico, 
and Guam. 

[2] DHS, National Infrastructure Protection Plan, Partnering to 
Enhance Protection and Resiliency (Washington, D.C.: January 2009). 
The NIPP is the document that articulates the risk management 
framework for DHS. The NIPP defines risk as a function of threat, 
vulnerability, and consequence. Threat is an indication of the 
likelihood that a specific type of attack will be initiated against a 
target. Vulnerability is the probability that a particular attempted 
attack will succeed against a particular target. Consequence is the 
effect of a successful attack. For more information on the NIPP, see 
GAO, Critical Infrastructure Protection: Update to National 
Infrastructure Protection Plan Includes Increased Emphasis on Risk 
Management and Resilience, [hyperlink, 
http://www.gao.gov/products/GAO-10-296] (Washington, D.C.: Mar. 5, 
2010). 

[3] According to the NIPP, risk assessments should be complete, 
reproducible, documented, and defensible. To be complete, the 
methodology should assess consequence, vulnerability, and threat for 
every defined risk scenario. To be reproducible, the methodology must 
produce comparable, repeatable results, and must minimize the number 
of subjective judgments. To be documented, the methodology and the 
assessment must clearly document what information is used and how it 
is synthesized to generate a risk estimate. To be defensible, the 
methodology must logically integrate its components and be free from 
significant errors or omissions. These core criteria are described in 
detail later in this report. 

[4] GAO, Risk Management: Further Refinements Needed to Assess Risks 
and Prioritize Protective Measures at Ports and Other Critical 
Infrastructure, [hyperlink, http://www.gao.gov/products/GAO-06-91] 
(Washington, D.C.: Dec. 15, 2005). 

[5] GAO, Maritime Security: DHS Progress and Challenges in Key Areas 
of Port Security, [hyperlink, http://www.gao.gov/products/GAO-10-940T] 
(Washington, D.C.: July 21, 2010). 

[6] GAO, Maritime Security: Varied Actions Taken to Enhance Cruise 
Ship Security, but Some Concerns Remain, [hyperlink, 
http://www.gao.gov/products/GAO-10-400] (Washington, D.C.: Apr. 9, 
2010). 

[7] [hyperlink, http://www.gao.gov/products/GAO-06-91]. 

[8] We did not review or validate the intelligence information 
incorporated in MSRAM. 

[9] Dr. Ted Lewis and Dr. Thomas J. Mackin, Naval Postgraduate School, 
Maritime Security Risk Analysis Model (MSRAM) Independent Verification 
and Validation (IV&V) Report, prepared for the Coast Guard Research 
and Development Center, March 2010. 

[10] A. Barret et al., Evaluation of U.S. Coast Guard Terrorism Risk 
and Decision Analysis Models and Processes for Port, Waterways and 
Coastal Security (Los Angeles: National Center for Risk and Economic 
Analysis of Terrorism Events, University of Southern California, 
2009). Established in 2004, CREATE is an interdisciplinary national 
research center based at the University of Southern California and 
funded by the U.S. Department of Homeland Security. CREATE is focused 
on risk and economic analysis of the United States and comprises a 
team of experts from across the country, including partnerships with 
numerous universities and research institutions. 

[11] National Research Council of the National Academies, Review of 
the Department of Homeland Security's Approach to Risk Analysis 
(Washington, D.C.: 2010). 

[12] See, for example, GAO, Managing for Results: Enhancing Agency Use 
of Performance Information for Management Decision Making, [hyperlink, 
http://www.gao.gov/products/GAO-05-927] (Washington, D.C.: Sept. 9, 
2005); Managing for Results: Challenges Agencies Face in Producing 
Credible Performance Information, [hyperlink, 
http://www.gao.gov/products/GAO/GGD-00-52] (Washington, D.C.: Feb. 4, 
2000); and The Results Act: An Evaluator's Guide to Assessing Agency 
Annual Performance Plans, [hyperlink, 
http://www.gao.gov/products/GAO/GGD-10.1.20] (Washington, D.C.: April 
1998). 

[13] OMB, Updated Principles for Risk Analysis, Memorandum M-07-Z4 
(Washington, D.C.: Sept. 19, 2007). 

[14] For example, the Homeland Security Act of 2002 (Pub. L. No. 107-
296, §201, 116 Stat. 2135, 2146 (2002)) requires DHS to perform risk 
assessments of key resources and critical infrastructure, and the 
Intelligence Reform and Terrorism Prevention Act of 2004 (Pub. L. No. 
108-458, §4001, 118 Stat. 3638, 3710 (2004)) requires that DHS's 
National Strategy for Transportation Security include the development 
of risk-based priorities across all transportation modes. 

[15] Pub. L. No. 107-295, 116 Stat. 2064, 2068-72 (2002). 

[16] Area Maritime Security Committees--with representatives from the 
federal, state, local, and private sectors--offer a venue to identify 
and deal with vulnerabilities in and around ports, as well as a forum 
for sharing information on issues related to port security. 

[17] Pub. L. No. 111-281, 124 Stat. 2905, 3004-05 (2010). In response 
to this statutory requirement, the Coast Guard developed the Industry 
Risk Analysis Model (IRAM). IRAM is a risk analysis tool that allows 
owners/operators of port facilities or vessels to perform a terrorism-
focused, security risk analysis of their property. The intended uses 
of IRAM results are to characterize the terrorism risk for a specific 
asset as evaluated by the owner/operator, provide a risk-based 
planning capability for updating facility or vessel operations plans, 
and provide a means to communicate risk between owners/operators and 
first responders. The Coast Guard expects to release IRAM in fall 2011. 

[18] The Coast Guard requires that at a minimum, MSRAM's list of 
targets should include all MTSA-regulated facilities, vessels, and 
barges. Department of Defense vessels and facilities, such as 
submarines, aircraft carriers, or naval bases, are optional in MSRAM. 
Smaller targets such as recreational boats, small commercial vessels, 
small waterside retail stores, and other targets with limited 
consequence potential should not be listed. 

[19] MSRAM does not address the use of the maritime transportation 
system as a means of transferring weapons or terrorists into the 
country. 

[20] MSRAM's risk assessment process asks users to evaluate each 
scenario considering the target's reasonable worst-case consequences. 
The Coast Guard defines this as the "maximum level of consequence for 
which there is at least a moderate likelihood of the attack mode being 
able to cause that damage level." 

[21] For impacts that are quantifiable in nature, such as numbers of 
fatalities, environmental spill sizes, and economic losses, the Coast 
Guard leveraged results from research for the monetization of impacts. 
For example, to monetize the death and injury determination, MSRAM 
uses $6.3 million per value of a statistical life, which is based on 
peer-reviewed research. 

[22] DHS, Transportation Systems Sector-Specific Plan, an Annex to the 
National Infrastructure Protection Plan (Washington, D.C.: 2010). This 
is the strategic plan for the sector implementing the requirements of 
Homeland Security Presidential Directive 7, Critical Infrastructure 
Identification, Prioritization, and Protection, and the requirements 
of the Intelligence Reform and Terrorism Prevention Act of 2004 (as 
amended) for the National Strategy for Transportation Security. The 
plan describes collaboratively developed strategies to reduce risks to 
critical transportation infrastructure from the broad range of known 
terrorism threats. 

[23] Lewis and Mackin, Maritime Security Risk Analysis Model (MSRAM) 
Independent Verification and Validation (IV&V) Report. 

[24] The Coast Guard defines an outlier as a score greater than two 
standard deviations from the national average for a given target class 
and scenario. 

[25] According to the Coast Guard, assessment and validation should 
not be limited to an annual cycle but should occur throughout the year 
as risk information changes. Sectors are expected to validate or 
update risk data annually. This can involve updating existing 
assessments if new information on a target is available or validating 
assessments if risk information is unchanged. 

[26] The Captain of the Port is the Coast Guard officer designated by 
the Commandant to enforce, within his or her respective area, port 
safety, security, and maritime environmental protection regulations, 
including, without limitation, regulations for the protection and 
security of vessels, harbors, and waterfront facilities. 

[27] Response capabilities are defined as the ability of the Coast 
Guard, port stakeholders and other governmental agencies to mitigate 
death/injury, primary economic, and environmental consequences of a 
successful attack. 

[28] [hyperlink, http://www.gao.gov/products/GAO-06-91]. 

[29] According to the Coast Guard, secondary economic impacts 
represent the expected follow-on economic effects of a successful 
attack. 

[30] IMPLAN stands for Impact Analysis for Planning. It is a tool that 
assesses economic relationships between primary and secondary economic 
impacts. 

[31] National Research Council, Department of Homeland Security 
Bioterrorism Risk Assessment: A Call for Change, Committee on 
Methodological Improvements to the Department of Homeland Security's 
Biological Agent Risk Analysis (Washington D.C.: 2008). 

[32] For more information on adaptive terrorist behavior, see A. Cox, 
"Some Limitations of "Risk = Threat x Vulnerability x Consequence" for 
Risk Analysis of Terrorist Attacks," Risk Analysis, vol. 28, no. 6 
(2008): 1749-1761. This article identifies potential limitations in 
the Risk = Threat x Vulnerability x Consequence formula for modeling 
risk that can undermine the ability of the model to guide resource 
allocations or optimize risk reductions. According to Cox, trying to 
directly assess probabilities for the actions of intelligent 
adversaries--instead of modeling how they adaptively pursue their 
goals--can produce ambiguous or mistaken risk estimates. 

[33] DHS, Risk Management Fundamentals: Homeland Security Risk 
Management Doctrine (Washington, D.C.: April 2011). 

[34] According to the Coast Guard, the Dynamic Risk Management Model 
is intended to enable decision makers to identify plausible future 
situations and identify robust risk management options with an 
adaptive adversary component. The Coast Guard is developing the 
prototype and expects to review and test the model over the next year. 

[35] For more information on network effects, see Gerald G. Brown, W. 
Matthew Carlyle, Javier Salmerón, and Kevin Wood, Operations Research 
Department, Naval Postgraduate School, Analyzing the Vulnerability of 
Critical Infrastructure to Attack and Planning Defenses (Monterey, 
Calif.: 2005). 

[36] DHS, Quadrennial Homeland Security Review Report: A Strategic 
Framework for a Secure Homeland (Washington, D.C.: February 2010). 

[37] National Research Council of the National Academies, Review of 
the Department of Homeland Security's Approach to Risk Analysis. 

[38] The MSRAM analysis processes focuses on evaluating terrorist 
attacks to individual targets, not systemwide attacks or simultaneous 
attacks on multiple maritime targets, which could lead to higher 
consequences. 

[39] The National Research Council of the National Academies states 
that because of the uncertainties inherent in terrorism risk analysis, 
it is crucial that DHS provide decision makers with complete 
information about how these uncertainties could affect decision 
making. See National Research Council of the National Academies, 
Review of the Department of Homeland Security's Approach to Risk 
Analysis. Also, DHS's Risk Management Fundamentals notes the 
importance of being transparent about methodology, limitations, and 
uncertainty so that decision makers have the most accurate, 
defensible, and practical information on which to base risk management 
decisions. 

[40] The Coast Guard conducted its verification and validation of 
MSRAM in 2009. The accreditation process began in February 2011. 
According to Coast Guard policy, MSRAM should have been internally 
accredited prior to implementation in 2005; however, officials 
explained that they did not conduct the accreditation of MSRAM prior 
to its initial deployment because of ongoing changes to the model. The 
officials added that MSRAM is now at an appropriate place in its 
development and evolution to be formally accredited. 

[41] National Research Council of the National Academies, Review of 
the Department of Homeland Security's Approach to Risk Analysis. 

[42] GAO, Homeland Security: Summary of Challenges Faced in Targeting 
Oceangoing Cargo Containers for Inspection, [hyperlink, 
http://www.gao.gov/products/GAO-04-557T] (Washington D.C., Mar. 31, 
2004). 

[43] Lewis and Mackin, Maritime Security Risk Analysis Model (MSRAM) 
Independent Verification and Validation (IV&V) Report. 

[44] Barret et al., Evaluation of U.S. Coast Guard Terrorism Risk and 
Decision Analysis Models and Processes for Port, Waterways and Coastal 
Security. 

[45] According to the Coast Guard, MSRAM information is not used 
exclusively in most cases, but is used in conjunction with other 
pertinent facts and factors to inform decision making. For example, 
MSRAM information, when combined with resource costs, national and 
Coast Guard strategic priorities, and Coast Guard legal mandates, is 
intended to be used to support risk-informed resource allocation 
decisions at the sector, district, area, and headquarters levels and 
risk-informed policy formulation at the national and area levels. 

[46] Operational activities include conducting boat escorts, 
implementing positive control measures--that is, stationing armed 
Coast Guard personnel in key locations aboard a vessel to ensure that 
the operator maintains control--and providing a security presence 
through various actions. 

[47] MSRO, formerly referred to as Operation Neptune Shield, are those 
operations conducted by the Coast Guard and its maritime partners to 
deny the use and exploitation of the maritime domain to criminal or 
hostile actors. Activities include vessel escorts, support to military 
outloads, periodic visits to maritime critical infrastructure and key 
resources, and security boardings, among other things. The required 
frequencies are considered classified. 

[48] The most recent MCIKR update occurred in spring 2011 and was 
informed by the past several years of MSRAM data, according to Coast 
Guard officials. 

[49] The Coast Guard's overarching goals for MSRAM include informing 
risk management decisions to preserve the marine transportation 
system; informing national, regional, and local security policy; and 
institutionalizing MSRAM throughout the Coast Guard as the security 
risk assessment and analysis program to support senior leadership 
decision-making, budget formulation, resource allocation, and 
performance measurement; among other goals. 

[50] For more information on the Port Security Grant Program, see GAO, 
Port Security Grant Program: Risk Model, Grant Management, and 
Effectiveness Measures Could Be Strengthened, [hyperlink, 
http://www.gao.gov/products/GAO-12-47] (Washington, D.C.: Nov. 17, 
2011). 

[51] The National Maritime Strategic Risk Assessment generates a 5-
year strategic maritime risk profile to serve as the basis for 
informing (1) the Coast Guard's budget and planning process, (2) the 
enterprise strategic planning direction, and (3) DHS's integrated 
planning process. The most recent plan came out in 2006. The National 
Maritime Terrorism Threat Assessment is prepared in coordination with 
the DHS Homeland Infrastructure Threat and Risk Analysis Center and 
addresses terrorist threats to the U.S. maritime domain and serves as 
a comprehensive analysis of maritime terrorist threats for Coast Guard 
to pass along critical information to local security partners. The 
most recent assessment was conducted in July 2010. According to Coast 
Guard officials, a new National Maritime Terrorism Threat Assessment 
is anticipated to be released in the fall of 2011. The Combating 
Marine Terrorism Strategic and Performance Plan, released in July 
2008, is the second iteration of the Coast Guard's strategy for the 
PWCS mission. It identifies essential roles and responsibilities for 
Coast Guard mission managers and senior operational commanders and 
informs DHS and partner agencies on how the Coast Guard will be 
addressing the security issues for which it is responsible. The next 
plan will be issued in 2015. 

[52] According to the 2010 DHS Transportation Systems Sector-Specific 
Plan, the Transportation Sector Security Risk Assessment tool is used 
to conduct modal security risk assessments for each of the primary 
transportation modes, as well as sub-modal groups, such as the school 
bus transportation system. 

[53] There are 18 critical infrastructure and key resource sectors 
addressed by the NIPP. They are (1) agriculture and food; (2) banking 
and finance; (3) chemical; (4) commercial facilities; (5) 
communications; (6) critical manufacturing; (7) dams; (8) defense 
industrial base; (9) emergency services; (10) energy; (11) government 
facilities; (12) healthcare and public health; (13) information 
technology; (14) national monuments and icons; (15) nuclear reactors, 
materials, and waste; (16) postal and shipping; (17) transportation 
systems; and (18) water. Each of these sectors has data in MSRAM 
except banking and finance, public health, emergency services, 
information technology, and postal and shipping. 

[54] In a May 2011 guidance document, the Coast Guard states that 
sector commanders should integrate MSRAM analysis into their 
operations to support risk communication, special security events, 
determining a daily security risk profile for a given area, port 
security exercises, risk management planning, Area Maritime Security 
Plans, and the Port Security Grant Program field review process, among 
other areas. 

[55] Area Maritime Security Plans have been established pursuant to 
the Maritime Transportation Security Act of 2002. Content requirements 
for the plans were established by 33 C.F.R. § 103.505 and expanded by 
the Security and Accountability For Every Port (SAFE Port) Act of 2006 
to include a Salvage Response Plan. The plans are intended to sponsor 
and support engagement with port community stakeholders to develop, 
test, and when necessary, implement joint efforts for responding to 
and mitigating the effects of a maritime transportation security 
incident. 

[56] The High Interest Vessel matrix is a risk-based tool used by 
Coast Guard sectors to evaluate the security risk of a vessel entering 
into port. According to DHS, sector staff use multiple databases and 
intelligence data to complete the matrix, focusing on security factors 
such as the vessel's size, cargo, operations, and security 
performance. Each of these security factors constitutes a component of 
the vessel's High Interest Vessel matrix score. When a vessel's score 
meets or exceeds a specific number, the Sector Commander designates 
the vessel as a high interest vessel and takes action to mitigate the 
risk it poses. 

[57] The Coast Guard expects personnel in each of its nine districts 
to maintain competency in MSRAM to assist sector personnel, and to 
support sectors in validating MSRAM data, among other things. 

[58] Coast Guard officials told us that MSRAM provides a static 
picture of risk at a given point in time, but maritime risk can be 
dynamic. For example, as a barge carrying dangerous cargo moves 
through a port area, the risk to assets in that port area changes 
based on the location of the barge. 

[59] According to the Coast Guard, the Alternatives Evaluation Module 
enables users to characterize the differences in their risk profile 
for a number of alternative environments, such as seasonal changes, 
technology changes, or changes in threat, consequence, or 
vulnerability. The Simplified Reporting Interface is designed to 
provide a simple interface for generating risk information tailored to 
support decision-making processes inside and outside the Coast Guard, 
and was designed to be used by sector staff who may only have a basic 
understanding of MSRAM's capabilities. The Daily Risk Profile allows 
users to identify which targets in their dataset are within their area 
of responsibility each day. The Risk Management Module provides the 
capability to develop and communicate risk mitigation strategies that 
the Coast Guard and its partners provide for every maritime scenario 
within the user's area of responsibility. 

[60] Officials from 11 sectors did not know or could not provide an 
answer on the usefulness of the Alternatives Evaluation Module. 

[61] Officials from 5 sectors did not know or could not provide an 
answer on the usefulness of the Simplified Reporting Interface. 

[62] The training focuses on understanding the basics of MSRAM, 
establishing or improving risk analysis and risk management skills, 
establishing or improving risk communication skills, and promoting 
risk management best practices, among other goals. 

[63] Additionally, efforts are under way to formalize MSRAM training, 
which begins with a training needs analysis. Officials added that the 
analysis will focus on how best to support MSRAM users in the field 
and will identify what a MSRAM user has to know or be able to do to 
use MSRAM. The analysis will also identify job skills needed for MSRAM 
users and will identify forms of support, such as job aids and 
training. 

[64] In the case of the 1 sector without a MSRAM specialist, their 
port security specialist had unexpectedly left the sector and the 
district-level port security specialist was managing the sector's 
MSRAM duties until a port security specialist could be hired. The 
sector officials reported that they expected to hire and train a new 
port security specialist before the end of calendar year 2011. 

[65] GAO, Standards for Internal Control in the Federal Government, 
[hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1] 
(Washington, D.C.: November 1999). 

[66] The Coast Guard does not include risk reduction efforts taken by 
private industry in the percentage of risk Coast Guard reduces. The 
portion of maritime risk subject to Coast Guard influence was 
estimated by Coast Guard officials in 2005. 

[67] Outcome measures describe the intended result of carrying out a 
program or activity. The Coast Guard has three additional risk 
reduction performance measures for its PWCS mission, which are 
considered subsets of the overall risk reduction measure: percentage 
reduction of maritime security risk resulting from Coast Guard 
consequence management, percentage reduction of maritime security risk 
resulting from Coast Guard efforts to prevent a terrorist entering the 
United States via maritime means, and the percentage reduction of 
maritime security risk resulting from Coast Guard efforts to prevent a 
weapon of mass destruction from entering the United States via 
maritime means. In addition, the Coast Guard has additional output and 
activity metrics to support performance evaluation in this mission 
area. 

[68] GAO, Department of Homeland Security: Progress Report on 
Implementation of Mission and Management Functions, [hyperlink, 
http://www.gao.gov/products/GAO-07-454] (Washington, D.C.: Aug. 17, 
2007). 

[69] Targets for fiscal years 2011 and 2012 have increased because of 
a revised methodology for estimating risk reduction that will take 
effect in fiscal year 2011. This revised methodology is discussed 
later in this section. 

[70] Information also comes from DHS's Risk Analysis Process for 
Informed Decision Making (RAPID) project, which is designed to provide 
strategic planning guidance and support resource allocation decisions 
at the DHS level. 

[71] According to Coast Guard officials, the subject matter experts 
are drawn from the three pillars of the PWCS mission: maritime domain 
awareness, regime, and MSRO. Although a reviewer of the model 
recommended using outside independent experts to help estimate risk 
reduction, Coast Guard uses only internal subject matter experts. 
Coast Guard officials explained that using external experts would be 
challenging because of the time needed to inform them about Coast 
Guard operations and activities. 

[72] According to DHS's Risk Management Fundamentals, elicitations 
involve using structured questions to gather information from 
individuals with in-depth knowledge of specific areas or fields. 

[73] The risk reduction model makes use of estimates of how often 
terrorists would be expected to attack maritime targets together with 
information from MSRAM on threats, vulnerabilities, and consequences. 
In addition, the Coast Guard has many activities that can reduce risk 
that are not focused on individual targets and are therefore not 
estimated in MSRAM, such as maritime regulations enforcement and 
vessel security boardings. These activities are factored into the risk 
reduction measure. The risk reduction measure also includes two attack 
scenarios--weapons of mass destruction and terrorist transfer 
scenarios--that are not included in MSRAM. 

[74] For example, see Barret et al., Evaluation of U.S. Coast Guard 
Terrorism Risk and Decision Analysis Models and Processes for Port, 
Waterways and Coastal Security. In addition, see M.E. Cutts, Improving 
the Coast Guard Ports, Waterways, and Coastal Security Outcome Measure 
(Monterey, Calif.: Naval Postgraduate School, June 2009). These 
reviews recommended a number of improvements, many of which the Coast 
Guard incorporated into the model. 

[75] According to the Coast Guard, in 2009 a total of 26 subject 
matter experts were used, mostly from headquarters. In 2010, a total 
of 46 subject matter experts were used coming from headquarters, 
areas, districts, sectors, and operational units. 

[76] [hyperlink, http://www.gao.gov/products/GAO/GGD-10.1.20]. 

[77] [hyperlink, http://www.gao.gov/products/GAO/GGD-00-52]. 

[78] The model utilizes judgments as well as actual Coast Guard patrol 
data, mapping data that show Coast Guard assets in relation to 
potential targets, as well as consequence models from MSRAM. 

[79] See Barret et al., Evaluation of U.S. Coast Guard Terrorism Risk 
and Decision Analysis Models and Processes for Port, Waterways and 
Coastal Security. This review examined the Coast Guard's risk 
reduction model before the model was upgraded and improved in 2010. 

[80] OMB, Updated Principles for Risk Analysis. 

[81] In February 2010, DHS issued its first Quadrennial Homeland 
Security Review report, outlining a strategic framework for homeland 
security to guide the activities of the department and its homeland 
security partners, including federal, state, local, and tribal 
government agencies; the private sector; and nongovernmental 
organizations. The report identified five homeland security missions--
Preventing Terrorism and Enhancing Security, Securing and Managing Our 
Borders, Enforcing and Administering Our Immigration Laws, 
Safeguarding and Securing Cyberspace, and Ensuring Resilience to 
Disasters--and goals and objectives to be achieved within each mission. 

[82] According to a DHS official, there are no other risk reduction 
measures similar to the Coast Guard's risk reduction measure chosen as 
strategic measures. 

[83] For fiscal year 2011, DHS identified 85 strategic measures for 
assessing its progress in achieving its Quadrennial Homeland Security 
Review missions and goals. In addition to these strategic measures, 
DHS also has 130 management measures, which DHS uses for resource 
allocation and other internal decision-making purposes, such as 
program evaluation. DHS includes its management measures in the 
Strategic Context presented to Congress with the department's budget 
request. 

[84] See [hyperlink, http://www.gao.gov/products/GAO-06-91]. 

[End of section] 

GAO’s Mission: 

The Government Accountability Office, the audit, evaluation, and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the 
performance and accountability of the federal government for the 
American people. GAO examines the use of public funds; evaluates 
federal programs and policies; and provides analyses, recommendations, 
and other assistance to help Congress make informed oversight, policy, 
and funding decisions. GAO’s commitment to good government is 
reflected in its core values of accountability, integrity, and 
reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO’s website [hyperlink, http://www.gao.gov]. Each 
weekday afternoon, GAO posts on its website newly released reports, 
testimony, and correspondence. To have GAO e mail you a list of newly 
posted products, go to [hyperlink, http://www.gao.gov] and select “E-
mail Updates.” 

Order by Phone: 

The price of each GAO publication reflects GAO’s actual cost of 
production and distribution and depends on the number of pages in the 
publication and whether the publication is printed in color or black 
and white. Pricing and ordering information is posted on GAO’s 
website, [hyperlink, http://www.gao.gov/ordering.htm]. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or 
TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card, 
MasterCard, Visa, check, or money order. Call for additional 
information. 

Connect with GAO: 

Connect with GAO on facebook, flickr, twitter, and YouTube.
Subscribe to our RSS Feeds or E mail Updates. Listen to our Podcasts.
Visit GAO on the web at www.gao.gov. 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 
Website: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]; 
E-mail: fraudnet@gao.gov; 
Automated answering system: (800) 424-5454 or (202) 512-7470. 

Congressional Relations: 

Ralph Dawn, Managing Director, dawnr@gao.gov, (202) 512-4400
U.S. Government Accountability Office, 441 G Street NW, Room 7125
Washington, DC 20548. 

Public Affairs: 
Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800
U.S. Government Accountability Office, 441 G Street NW, Room 7149 
Washington, DC 20548.