Federal Air Marshal Service:

Actions Needed to Better Incorporate Risk in Deployment Strategy

GAO-16-582: Published: May 31, 2016. Publicly Released: May 31, 2016.

Additional Materials:

Contact:

Jennifer Grover
(202) 512-7141
groverj@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

What GAO Found

The U.S. Federal Air Marshal Service (FAMS) deploys air marshals in part based on assessed risk, but could better incorporate risk in its deployment strategy. FAMS may deploy air marshals on flights with known risk—meaning certain higher risk flights where the Transportation Security Administration (TSA) or FAMS knows of an increase in the threat or consequence of a terrorist attack—as well as on other international and domestic flights of U.S. air carriers. However, GAO identified three ways FAMS could better incorporate risk into its deployment decisions, in accordance with Department of Homeland Security (DHS) and TSA risk management policy and guidance.

  • FAMS considers its travel budget and number of personnel, but not risk when initially dividing its annual resources between domestic and international flights. With this approach, FAMS attempts to maximize the total number of flights it can cover, but because this model does not account for risk, FAMS cannot ensure it is devoting its resources to the highest risk flights overall.
  • FAMS officials report that when FAMS revised its domestic deployment strategy in 2014, their choice of geographic focus areas and resource allocation levels were based on subject matter experts' professional judgment, not a risk assessment. FAMS officials stated this approach was appropriate because they were updating the strategy, not conducting a study. While providing perceptions of risk, the experts' input was not systematically collected and assigned risk values. Doing so would better position FAMS to ensure its resources are targeted using a risk-based approach.
  • FAMS does not document rationales for its international deployment decisions because it has not identified a need to do so. Without documentation of the basis for these decisions, neither FAMS nor an external party can effectively oversee these decisions.

The performance measure FAMS has used to assess the extent to which it has met its flight coverage and resource use targets—the TSA coverage score—reflects some but not all attributes of successful performance measures. The TSA coverage score is a composite score for 11 performance categories, including air marshal coverage of certain domestic and international flights. The measure lacks clarity because the measure's name and definition are not consistent with the methodology used to calculate it. For example, the measure aggregates flight coverage and resource allocation information, which reduces the clarity of the score and makes it difficult to interpret. It also lacks objectivity because, as a composite measure, it does not show performance below or above desired levels in the 11 categories—information that would aid decision making. FAMS officials stated that they did not report scores for all categories to TSA or FAMS leadership because they were not asked to do so. Without clear and objective performance information, DHS, TSA, and FAMS leadership may be making decisions based on an inaccurate impression of FAMS's performance. This is a public version of a classified report GAO issued in February 2016. Information that DHS deemed classified or sensitive has been omitted.

Why GAO Did This Study

Following the September 11, 2001, hijacking of four U.S. airliners, individuals with terrorist ties have attempted attacks against the nation's civil aviation system. To help address such threats, FAMS, an office within TSA, is tasked with promoting confidence in the nation's civil aviation system through the deployment of air marshals to protect U.S. air carriers, airports, passengers, and crews. GAO was asked to review FAMS operations. This report examines the extent to which (1) FAMS deploys air marshals based on risk and (2) FAMS's performance measure reflects attributes of successful performance measures. GAO reviewed FAMS's strategy and performance measure documents, analyzed FAMS's fiscal years 2010 through 2014 flight coverage and performance measure data, and interviewed FAMS and TSA officials.

What GAO Recommends

GAO recommends that FAMS (1) further incorporate risk into FAMS's method for dividing resources between international and domestic flights, (2) conduct a risk assessment to support certain domestic deployment decisions, (3) document the rationale for FAMS's selection of international deployment destinations, (4) adopt a consistent name and definition for the TSA coverage score, and (5) report performance results for all categories that comprise the score. DHS concurred with all of the recommendations.

For more information, contact Jennifer Grover at (202) 512-7141 or groverj@gao.gov.

Recommendations for Executive Action

  1. Status: Open

    Comments: In May 2016, we found that FAMS officials considered risk when selecting specific domestic and international flights to cover, but they did not consider risk when deciding how to initially divide their annual resources between domestic and international flights. Rather, each year FAMS considered two variables--travel budget and number of air marshals--to identify the most efficient way to divide the agency's resources between domestic and international flights. As a result, we recommended that FAMS incorporate risk into FAMS's method for initially setting its annual target numbers of average daily international and domestic flights to cover. In March 2017, TSA officials reported that FAMS was continuing to identify ways to refine the methodology FAMS uses to allocate resources between international and domestic flights. Specifically, TSA officials noted that FAMS was considering ways to incorporate information on the travel patterns of known or suspected terrorists, trends in TSA PreCheck passenger data, airport screening capabilities, and other factors. FAMS officials also reported that, as part of this effort, they were reviewing their International Concept of Operations. It is unclear how these steps will address the recommendation. To fully address this recommendation, FAMS should incorporate risk into its method for initially setting its annual target numbers of average daily international and domestic flights to cover.

    Recommendation: To better ensure that FAMS uses its resources to cover the highest-risk flights, in addition to considering risk when determining how to divide FAMS's international flight coverage resources among international destinations, the Director of FAMS should incorporate risk into FAMS's method for initially setting its annual target numbers of average daily international and domestic flights to cover.

    Agency Affected: Department of Homeland Security: Transportation Security Administration: Office of Law Enforcement - Federal Air Marshal Service

  2. Status: Open

    Comments: In May 2016, we reported that FAMS's choice of domestic geographic focus areas and resource allocation levels were based on professional judgment, not risk assessment. With regard to the geographic focus areas, for example, FAMS officials explained that they did not conduct a risk assessment to inform this decision, but rather selected these areas in consultation with 30 subject matter experts from various offices within TSA based on their intuitive, qualitative perceptions of threats, vulnerabilities, potential impacts, history, and the demographics of the areas. Without fully incorporating risk when determining such priorities, FAMS cannot reasonably ensure it is targeting its resources to the highest-risk flights. As a result, we recommended that FAMS conduct and document a risk assessment--systematically collecting information on and assigning value to current risks--to further support FAMS's domestic resource allocation decisions, including the identification of high-priority geographic areas. In March 2017, TSA officials explained that they were continuing to develop their "risk-by-flight" initiative--a long-term effort to develop a method of assigning each domestic flight a relative risk score to assist in identifying high-risk flights. At the time of our report in 2016, FAMS officials estimated that the risk-by-flight tool would probably be ready for use within 7 to 10 years. In March 2017, TSA officials stated that they had developed a prototype Risk-Based Resource Deployment Decision Aid, which they refer to as R2D2. TSA officials further reported that the DHS Science and Technology Directorate had contracted for the development of a risk engine--based on the R2D2 data--to assign risk values to all U.S.-carrier domestic and international flights. TSA officials reported that this contract runs through early 2018. To fully address this recommendation, FAMS should conduct and document a risk assessment to further support FAMS's domestic resource allocation decisions, including the identification of high-priority geographic areas.

    Recommendation: To better ensure that FAMS uses its resources to cover the highest-risk flights, the Director of FAMS should conduct and document a risk assessment--systematically collecting information on and assigning value to current risks--to further support FAMS's domestic resource allocation decisions, including the identification of high-priority geographic areas.

    Agency Affected: Department of Homeland Security: Transportation Security Administration: Office of Law Enforcement - Federal Air Marshal Service

  3. Status: Closed - Implemented

    Comments: In May 2016, we reported that the U.S. Federal Air Marshal Service (FAMS) did not document rationales for its international deployment decisions. FAMS officials knowledgeable about FAMS's deployment strategy reported that they selected international destinations to cover and the proportion of flights to cover in each destination based upon intelligence and vulnerability information. These officials reported that they documented the results of these decisions--the number and percentage of U.S.-flagged air carrier flights air marshals will cover by country and specific airport--in a planning roster, but they did not document the rationales for these decisions because they had not identified a need to do so. DHS's Integrated Risk Management Framework establishes transparency and documentation as important characteristics of homeland security risk management. The framework also states that documentation enables critical analyses of the approach. Consequently, we recommended that the Director of FAMS document the rationale for FAMS's selection of international destinations for air marshal deployment and the proportion of flights to cover at each destination. FAMS officials report that in response to GAO's report, in the summer of 2016, FAMS began documenting on a monthly and quarterly basis the rationales for its selection of international destinations to cover and the proportion of flights to cover in each destination. A senior FAMS official reported that this tool has been helpful as it permits FAMS officials to analyze deployment changes over time. Further, documentation of the basis for these decisions also permits external parties with authorized oversight responsibilities to determine the extent to which the decisions are intelligence-driven and risk-based, and management's directives are being carried out. As a result, this recommendation is closed as implemented.

    Recommendation: To better ensure that FAMS uses its resources to cover the highest-risk flights, the Director of FAMS should document the rationale for FAMS's selection of international destinations for air marshal deployment and the proportion of flights to cover at each destination.

    Agency Affected: Department of Homeland Security: Transportation Security Administration: Office of Law Enforcement - Federal Air Marshal Service

  4. Status: Closed - Implemented

    Comments: In May 2016, we reported that the performance measure FAMS has used to assess the extent to which it has met its flight coverage and resource use targets--previously referred to as the TSA coverage score--reflected some but not all attributes of successful performance measures. The TSA coverage score was a composite score for 11 performance categories, including air marshal coverage of certain domestic and international flights, but the measure lacked clarity because the measure's name and definition were not consistent with the methodology used to calculate it. Further, we found that FAMS and TSA used different names for the score--FAMS referred to it internally as the TSA coverage score and TSA referred to it more formally as "the percentage of risk-based flight coverage by the FAMS." As a result, we recommended that FAMS adopt a consistent name and definition for this performance measure that accurately reflects its calculation method and composite nature. In March 2017, TSA officials reported that FAMS had changed the name of the measure to "Composite Index of Federal Air Marshal Service Risk-Based Flight Coverage Goals" and provided documentation showing that FAMS and TSA now referred to this measure with the same name. These changes improve the clarity of the measure to users, including DHS and TSA leadership, who rely on the measure to help them oversee and assess FAMS's performance. As a result, this recommendation is closed as implemented.

    Recommendation: To improve the usefulness of the performance information DHS, TSA, and FAMS leadership use to oversee FAMS performance in achieving its mission priorities, the Director of FAMS should adopt a consistent name and definition for the performance measure referred to as the TSA coverage score that accurately reflects its calculation method and composite nature.

    Agency Affected: Department of Homeland Security: Transportation Security Administration: Office of Law Enforcement - Federal Air Marshal Service

  5. Status: Closed - Implemented

    Comments: In May 2016, we reported that TSA and FAMS leadership did not have visibility into FAMS's performance in most of the 11 categories that comprise the TSA coverage score and TSA officials who reviewed FAMS's performance reported that this would be useful. In March 2017, TSA officials reported that FAMS now reports the performance results for each of the 11 subcategories of the TSA coverage score--now referred to as the Composite Index of Federal Air Marshal Service Risk-Based Flight Coverage Goals--to both FAMS and TSA leadership on a regular basis. Specifically, FAMS officials provided documentation showing that they now include results for the 11 categories in monthly reports to the Director of FAMS and in quarterly reports to the TSA Office of the Chief Financial Officer. With more complete information on FAMS performance, DHS, TSA, and FAMS leadership are now better positioned to make decisions based on a more accurate impression of FAMS's performance and identify possible corrective action where FAMS's performance is below targets. As a result, this recommendation is closed as implemented.

    Recommendation: To improve the usefulness of the performance information DHS, TSA, and FAMS leadership use to oversee FAMS performance in achieving its mission priorities, the Director of FAMS should report the performance results for each of the subcategories that comprise the TSA coverage score to FAMS and TSA leadership.

    Agency Affected: Department of Homeland Security: Transportation Security Administration: Office of Law Enforcement - Federal Air Marshal Service

 

Explore the full database of GAO's Open Recommendations »

Oct 17, 2017

Oct 11, 2017

Oct 4, 2017

Sep 27, 2017

Sep 26, 2017

Sep 25, 2017

Sep 12, 2017

Sep 11, 2017

Sep 7, 2017

Looking for more? Browse all our products here