Aviation Security:
Actions Needed to Systematically Evaluate Cost and Effectiveness Across Security Countermeasures
GAO-17-794: Published: Sep 11, 2017. Publicly Released: Sep 11, 2017.
Additional Materials:
- Highlights Page:
- Full Report:
- Accessible Version:
Contact:
(202) 512-7141
groverj@gao.gov
Office of Public Affairs
(202) 512-4800
youngc1@gao.gov
What GAO Found
The Transportation Security Administration (TSA) has data on the effectiveness of some, but not all of its passenger aviation security countermeasures. Specifically, TSA has data on passenger prescreening, checkpoint and checked baggage screening, and explosives detection canines. Further, TSA is taking steps to improve the quality of this information. However, it does not have effectiveness data for its Behavior Detection and Analysis (BDA) program and the U.S. Federal Air Marshal Service (FAMS). For BDA—a program to identify potential threats by observing passengers for behaviors indicative of stress, fear, or deception—in July 2017, GAO reported that (1) TSA does not have valid evidence supporting most of its behavioral indicators, and (2) TSA should continue to limit future funding for its behavior detection activities until it can provide such evidence. For FAMS—a program that deploys armed law enforcement officers on certain flights at an annual cost of about $800 million for fiscal year 2015—officials reported that one of the primary security contributions is to deter attacks. However, TSA does not have information on its effectiveness in doing so, nor does it have data on the deterrent effect resulting from any of its other aviation security countermeasures. While officials stated that deterrence is difficult to measure, the Government Performance and Results Act of 1993, as updated, provides that agencies are to assess the effectiveness of their programs. Further, the Office of Management and Budget and GAO have suggested approaches for measuring deterrence. Developing such methods for TSA countermeasures, especially for an effort such as FAMS in which the primary goal is deterrence, would enable TSA to determine whether its substantial investment is yielding results.
TSA has a tool to compare the security effectiveness of some aviation security countermeasures, but has no efforts underway to systematically evaluate potential cost and effectiveness tradeoffs across all countermeasures. In 2014, the agency developed a tool to analyze the security effectiveness of alternate combinations of some countermeasures for the purpose of informing acquisition and deployment decisions, but does not have a tool to assess such tradeoffs across the entire system of countermeasures. TSA officials explained that the aviation security system is constantly evolving, and assessing a system in flux is challenging. However, DHS policy and TSA's strategic plan call for the systematic evaluation of costs and effectiveness of TSA's chosen mix of aviation security countermeasures. Without such an analysis, TSA is not well positioned to strike an appropriate balance of costs, effectiveness, and risk.
This is a public version of a classified report that GAO issued in August 2017. Information that TSA deemed classified or sensitive security information, such as the results of TSA's covert testing and details about TSA's screening procedures, have been omitted.
Why GAO Did This Study
Since the attacks of September 11, 2001, TSA has spent billions of dollars on aviation security programs. However, recent attacks involving aircraft and airports in other countries underscore the continued threat to aviation and the need for an effective aviation security program.
GAO was asked to review TSA's passenger aviation security countermeasures. This report examines the extent to which TSA has (1) information on the effectiveness of selected passenger aviation security countermeasures and (2) systematically analyzed the cost and effectiveness tradeoffs among countermeasures.
GAO reviewed TSA documentation on the effectiveness of six passenger aviation security countermeasures in fiscal year 2015—the most recent year for which data were available. GAO selected these countermeasures because they involve direct interaction with passengers, their belongings, or their personal information, and are largely operated and funded by TSA. GAO also reviewed TSA documents and interviewed TSA officials regarding efforts to systematically analyze cost and effectiveness tradeoffs across countermeasures.
What GAO Recommends
GAO recommends that TSA (1) explore and pursue methods to assess the deterrent effect of TSA's passenger aviation security countermeasures, with FAMS as a top priority to address, and (2) systematically evaluate the potential cost and effectiveness tradeoffs across aviation security countermeasures. DHS concurred with these recommendations.
For more information, contact Jennifer Grover at (202) 512-7141 or groverj@gao.gov.
Recommendations for Executive Action
Status: Open
Comments: DHS concurred with this recommendation and has begun taking steps intended to address it. In May 2018, officials with DHS's Office of Requirements and Capabilities Analysis (ORCA) reported conducting a literature review to identify ways DHS might be able to measure deterrence. ORCA officials reported looking to published studies and other agencies to identify data sources and methods, and are currently in the process of developing a model to assess the deterrent value of various aviation security countermeasures. To fully address this recommendation, TSA will need to fully pursue such methods and demonstrate a particular focus on capturing the deterrent effect of Federal Air Marshals.
Recommendation: The Administrator of TSA should explore and pursue methods to assess the deterrent effect of TSA's passenger aviation security countermeasures; such an effort should identify FAMS—a countermeasure with a focus on deterring threats—as a top priority to address. (Recommendation 1)
Agency Affected: Department of Homeland Security: Transportation Security Administration
Status: Open
Comments: DHS officials concurred with this recommendation and have begun taking steps intended to address it. Specifically, in May 2018, TSA officials reported that they were in the process of enhancing their annual Resource Allocation Planning (RAP) process with additional consideration of the costs and effectiveness of aviation security countermeasures. According to TSA officials, they piloted an early version of this enhanced RAP process in 2017 and are looking to more fully adopt this approach in the fiscal year 2021 RAP process occurring in late 2018 and early 2019. To address the intent of our recommendation, TSA will need to demonstrate that they have evaluated the costs and effectiveness of individual aviation security countermeasures and then used this information to systematically consider the potential cost and effectiveness tradeoffs across countermeasures. Agency officials anticipate fully addressing this recommendation by April 2019.
Recommendation: The Administrator of TSA should systematically evaluate the potential cost and effectiveness tradeoffs across countermeasures, as TSA improves the reliability and extent of its information on the effectiveness of aviation security countermeasures. (Recommendation 2)
Agency Affected: Department of Homeland Security: Transportation Security Administration
Explore the full database of GAO's Open Recommendations
»
Feb 7, 2019
-
Presidential Security:
Vetting of Individuals and Secure Areas at Mar-a-LagoGAO-19-246: Published: Jan 23, 2019. Publicly Released: Feb 7, 2019.
Feb 5, 2019
-
Presidential Travel:
Secret Service and DOD Need to Ensure That Expenditure Reports Are Prepared and Submitted to CongressGAO-19-178: Published: Jan 17, 2019. Publicly Released: Feb 5, 2019.
Jan 8, 2019
-
Federal Protective Service:
DHS Should Take Additional Steps to Evaluate Organizational PlacementGAO-19-122: Published: Jan 8, 2019. Publicly Released: Jan 8, 2019.
Dec 19, 2018
-
Critical Infrastructure Protection:
Actions Needed to Address Significant Weaknesses in TSA's Pipeline Security Program ManagementGAO-19-48: Published: Dec 18, 2018. Publicly Released: Dec 19, 2018.
Dec 13, 2018
-
National Security:
Long-Range Emerging Threats Facing the United States As Identified by Federal AgenciesGAO-19-204SP: Published: Dec 13, 2018. Publicly Released: Dec 13, 2018.
Dec 6, 2018
-
2017 Disaster Contracting:
Action Needed to Better Ensure More Effective Use and Management of Advance ContractsGAO-19-93: Published: Dec 6, 2018. Publicly Released: Dec 6, 2018.
Nov 29, 2018
-
Coast Guard Acquisitions:
Addressing Key Risks Is Important to Success of Polar Icebreaker ProgramGAO-19-255T: Published: Nov 29, 2018. Publicly Released: Nov 29, 2018.
Nov 28, 2018
-
Aviation Security:
TSA Uses a Variety of Methods to Secure U.S.-bound Air Cargo, but Could Do More to Assess Their EffectivenessGAO-19-162: Published: Nov 28, 2018. Publicly Released: Nov 28, 2018.
Oct 11, 2018
-
Coast Guard Acquisitions:
Lessons Learned to Inform Coast Guard and NOAA Shipbuilding EffortsGAO-19-147T: Published: Oct 11, 2018. Publicly Released: Oct 11, 2018.
Sep 21, 2018
-
Chemical Terrorism:
A Strategy and Implementation Plan Would Help DHS Better Manage Fragmented Chemical Defense Programs and ActivitiesGAO-18-562: Published: Aug 22, 2018. Publicly Released: Sep 21, 2018.
Looking for more? Browse all our products here