Aviation Security: TSA Should Ensure Testing Data Are Complete and Fully Used to Improve Screener Training and Operations
Highlights
What GAO Found
The Transportation Security Administration (TSA) uses a variety of programs to train and evaluate Transportation Security Officers (TSO) who are responsible for screening passengers and baggage for threats to aviation security. For example, by law, TSOs must complete 40 hours of classroom training, 60 hours of on-the-job training, and certification tests before performing screening. Once certified, TSA requires TSOs to complete annual training under the National Training Plan. Since 2013, TSA has been phasing in a program to evaluate its training to inform use of training resources. TSA expects that this evaluation program should help the agency determine how well training meets TSOs' needs, provides them with needed knowledge and skill, and has an impact on their performance.
TSA measures TSO performance in various ways, including (1) annual proficiency reviews, which certify TSOs by evaluating their ability to carry out screening standard operating procedures; (2) assessments of X-ray machine operators' ability to identify prohibited items by displaying fictional threat items, such as guns or explosives, onto X-ray images of actual baggage; and (3) covert testing programs that use role players to take prohibited items through screening checkpoints to test TSOs or determine how TSOs interact with the public, among other things. Over the time periods GAO reviewed, TSA data on the results of annual proficiency reviews and covert testing on how TSOs interact with the public show that TSOs' scores (pass rates) varied by airport security risk category. GAO is not providing TSOs' scores for annual proficiency reviews, X-ray machine operator assessments, or covert testing for prohibited items at checkpoints in this report due to the sensitive or classified nature of the data or the data reliability concerns discussed below.
TSA has made use of annual proficiency review data to enhance TSO training, but its use of other testing data is constrained by incomplete and unreliable data. Specifically, due to software compatibility issues and a lack of automatic uploading capability, airport reporting on assessments of X-ray machine operators was not complete, as required by TSA policy, for each year of data GAO examined (fiscal years 2009 through 2014), limiting their reliability and use to enhance TSO training. In addition, for the data it does collect on these assessments, TSA has not taken steps to analyze these data nationwide, which could help the agency identify potential trends or opportunities to improve TSO performance. Furthermore, in 2015, TSA determined that prior year results of one of its two covert testing programs to assess TSOs' ability to identify prohibited items at checkpoints were unreliable, resulting in pass rates that were likely higher than actual TSO performance. TSA has since taken steps to enhance reliability by hiring a contractor to perform independent validation testing, among other things. Finally, TSA does not require or track implementation by field personnel of national recommendations related to these covert tests, thereby limiting the agency's ability to take advantage of the corrective actions identified from the tests.
Why GAO Did This Study
TSA trains TSOs to screen passengers and baggage for items that could pose a threat at nearly 440 airports across the country. One way TSA and the Department of Homeland Security (DHS) Office of Inspector General (OIG) measure TSO performance is through covert testing of TSA screening operations. In response to the findings from recent DHS OIG covert testing, the Secretary of DHS directed TSA in June 2015 to conduct further training for all TSOs and supervisors. GAO was asked to review TSA's efforts to train and test TSOs. This report examines (1) how TSA trains TSOs and evaluates the training; (2) how TSA measures TSO performance and what the data show; and (3) to what extent TSA uses TSO performance data to enhance TSO performance. GAO analyzed TSO performance data from 2009 through 2015, reviewed documents regarding TSA training and testing, and interviewed TSA officials at headquarters and 10 airports. GAO selected these airports based on airport risk categories, among other things. Information from these airports was not generalizable, but provided insights into TSO training and testing. This is a public version of a sensitive report that GAO issued in May 2016.
Recommendations
GAO recommends that TSA (1) collect complete data on assessments of X-ray machine operators, (2) analyze these data nationally for opportunities to enhance TSO performance, and (3) track the implementation of covert testing recommendations. TSA concurred with the recommendations.
Recommendations for Executive Action
| Agency Affected | Recommendation | Status |
|---|---|---|
| Department of Homeland Security | To improve TSA's ability to take full advantage of testing results to inform and potentially improve screening operations, the Secretary of the Department of Homeland Security should direct the Administrator of TSA to ensure that TSA officials at individual airports submit complete Threat Image Projection (TIP) results to the TSA national database as required, including manually submitting data when automated uploading is not available. |
Closed – Implemented
In May 2016, we reported on the Transportation Security Administration's (TSA) efforts to train and test its Transportation Security Officers (TSO). During the course of our review, we found that, due to software compatibility issues and a lack of automatic uploading capability, airport reporting on Threat Image Projection (TIP) assessments of X-ray machine operators (testing operators by projecting threat images onto real passenger baggage) was not being submitted by individual airports to the TSA national database, as required by TSA policy, for each year of data we examined. Due to this incomplete data, TSA was limited in its ability to use these data to enhance TSO training. Consequently, we recommended that TSA ensure that its personnel at individual airports submit complete TIP results to the TSA national database as required, including manually submitting TIP data when automated uploading is not available. In October 2016, TSA issued a TIP Operations Directive and Threat Image Projection Handbook which provided new procedures for TIP deployment and performance data collection and submission. In November 2016, TSA reported that all airports had received the operations directive and handbook to ensure TIP data are provided for all airports. Further, TSA stated that all TIP data non-compliance issues would be addressed by the Office of Security Operations through the Regional Managers. Subsequently, in January 2017, TSA provided examples of the required system-wide monthly reports, sent to all its regions for the month of October 2016, identifying airports with missing TIP data. TSA also provided examples of follow-up e-mails sent by Region 7 TSA officials to specific airports in order to ascertain the reasons for their missing data (in the October TIP report) and to obtain that data if possible. In May 2017, TSA provided additional information on the resolution of the 4 Region 7 airports with missing TIP data to show that data for 3 of the airports had been successfully obtained with the lone remaining airport not reporting TIP data due to lack of passenger volume during the time period. As a result of these efforts by TSA to identify and obtain or explain missing TIP data, this recommendation is closed as implemented.
|
| Department of Homeland Security | To improve TSA's ability to take full advantage of testing results to inform and potentially improve screening operations, the Secretary of the Department of Homeland Security should direct the Administrator of TSA to conduct analysis of national TIP data for trends that could inform training needs and improve future training and TSO performance assessments. |
Closed – Implemented
In May 2016, we reported on the Transportation Security Administration's (TSA) efforts to train and test its Transportation Security Officers (TSO). During the course of our review, we found that TSA had not taken steps to analyze the nationwide data from Threat Image Projection (TIP) assessments of X-ray machine operators (testing operators by projecting threat images onto real passenger baggage) which could help the agency identify potential trends or opportunities to improve TSO performance. Consequently, we recommended that TSA conduct analysis of national TIP data for trends that could inform training needs and improve future training and TSO performance assessments. In July 2017, TSA reported that it had analyzed several months of TIP data and determined that, while an effective vigilance tool for the TSO workforce in the operational checkpoint screening environment, TIP does not provide the type of information that can effectively contribute to the design or update of training for TSOs since it is based on the random introduction of threat images into passengers' bags. Instead, TSA reported that it will rely on current threat and intelligence information, along with TSO performance information gleaned from covert testing results and focused image interpretation training sessions to update and improve TSO training. Despite concluding that TIP results were not ideal for informing TSO training, we believe that TSA's analysis resulting in this determination was consistent with the intent of our recommendation. Therefore, we are closing this recommendation as implemented.
|
| Department of Homeland Security | To improve TSA's ability to take full advantage of testing results to inform and potentially improve screening operations, the Secretary of the Department of Homeland Security should direct the Administrator of TSA to track implementation by airports of Aviation Screening Assessment Program (ASAP) recommendations to ensure that corrective actions identified through ASAP testing are being applied. |
Closed – Implemented
In May 2016, we reported on the Transportation Security Administrations (TSA) efforts to train and test its Transportation Security Officers (TSO). During the course of our review, we found that TSA does not require or track implementation by field personnel of national recommendations made to airports through its Aviation Screening Assessment Program (ASAP). ASAP is a covert testing program designed to assess TSOs ability to identify prohibited items at airport checkpoints. As a result of not tracking implementation by field personnel at the airports, the agency's ability to ensure that corrective actions were being taken at airports to improve TSO performance was limited. Moreover, tracking the implementation of its recommendations, including the extent to which identified corrective actions are improving future TSO performance and test results, would help TSA better determine the extent to which its implemented recommendations are leading to improvements in screening operations and appropriately addressing identified root causes for previous test failures. Consequently, we recommended that TSA track implementation by airports of ASAP recommendations to ensure that corrective actions identified through ASAP testing are being applied. In January 2017, TSA provided additional information regarding the ASAP program and the incorporation of the covert testing done under ASAP into the new Security Effectiveness 7-Step Performance Improvement program. This new program incorporates the covert testing formerly done under ASAP with resulting Threat Detection Improvement Plans (TDIP), which are based on recommended actions stemming from each airports covert testing results. TSAs Office of Security Operations (OSO) now monitors airports progress against these plans in order to ensure that they are taking the necessary actions to improve TSO performance deficiencies identified during ASAP covert testing. In its January 2017 response, TSA also provided examples of improvement plans provided to OSO by two airports which included actions taken by those airports to address deficiencies identified in the covert tests. The document included airport responses to queries from OSO regarding their progress. These actions should help provide TSA with reasonable assurance that the needed corrective actions identified through its covert testing program are being applied as well as assess the extent to which corrective actions are improving TSO performance and future test results. As a result, this recommendation is closed as implemented.
|