Aviation Security: TSA Should Ensure Screening Technologies Continue to Meet Detection Requirements after Deployment
Fast Facts
TSA tests its screening technologies before installing them at airports to ensure that they are detecting certain dangerous items as intended.
But screening technologies can become less effective over time, and we found that TSA does not continue to fully test them once they are installed. Some airport equipment that detects trace explosives or tests bottled liquids wasn’t performing as intended when the Department of Homeland Security evaluated it in 2015 and 2016.
We recommended that TSA ensure that its screening technologies continue to meet requirements after they are installed at airports.
Airport screening equipment
Highlights
What GAO Found
The Department of Homeland Security's (DHS) Transportation Security Administration (TSA) operationalizes, or puts into effect, detection standards for its screening technologies by acquiring and deploying new technologies, which can take years. Detection standards specify the prohibited items (e.g., guns, explosives) that technologies are to detect, the minimum rate of detection, and the maximum rate at which technologies incorrectly flag an item. TSA operationalizes standards by adapting them as detection requirements, working with manufacturers to develop and test new technologies (software or hardware), and acquiring and deploying technologies to airports. For the standards GAO reviewed, this process took 2 to 7 years, based on manufacturers' technical abilities and other factors.
TSA's deployment decisions are generally based on logistical factors and it is unclear how risk is considered when determining where and in what order technologies are deployed because TSA did not document its decisions. TSA considers risks across the civil aviation system when making acquisition decisions. However, TSA did not document the extent risk played a role in deployment, and could not fully explain how risk analyses contributed to those decisions. Moving forward, increased transparency about TSA's decisions would better ensure that deployment of technologies matches potential risks.
Technology performance can degrade over time; however, TSA does not ensure that technologies continue to meet detection requirements after deployment to airports. TSA certifies technologies to ensure they meet requirements before deployment, and screeners are to regularly calibrate deployed technologies to demonstrate they are minimally operational. However, neither process ensures that technologies continue to meet requirements after deployment. In 2015 and 2016, DHS tested a sample of deployed explosives trace detection and bottled liquid scanner units and found that some no longer met detection requirements. Developing and implementing a process to ensure technologies continue to meet detection requirements after deployment would help ensure that TSA screening procedures are effective and enable TSA to take corrective action if needed.
Transportation Security Administration's (TSA) Process for Acquiring Screening Technologies to Meet Detection Standards
Why GAO Did This Study
TSA is responsible for overseeing security operations at roughly 440 TSA-regulated airports as part of its mission to protect the nation's civil aviation system. TSA uses technologies to screen passengers and their bags for prohibited items.
The TSA Modernization Act includes a provision for GAO to review TSA's deployment of screening technologies, and GAO was asked to review the detection standards of these screening technologies. This report addresses, among other things, (1) how TSA operationalizes detection standards, (2) the extent to which TSA considered risk when making deployment decisions, and (3) the extent to which TSA ensures technologies continue to meet detection requirements after deployment.
GAO reviewed DHS and TSA procedures and documents, including detection standards; visited DHS and TSA testing facilities; observed the use of screening technologies at seven airports, selected for varying geographic locations and other factors; and interviewed DHS and TSA headquarters and field officials.
Recommendations
GAO is making five recommendations, including that TSA document analysis of risk in deploying technologies, and implement a process to ensure technologies continue to meet detection requirements after deployment. DHS agreed with all five recommendations and said TSA either has taken or will take actions to address them.
Recommendations for Executive Action
| Agency Affected | Recommendation | Status |
|---|---|---|
| Transportation Security Administration | The TSA Administrator should update TSA guidance for developing and approving screening technology explosives detection standards to reflect designated procedures, the roles and responsibilities of stakeholders, and changes in the agency's organizational structure. (Recommendation 1) |
Closed – Implemented
In December 2019, we reported on the Transportation Security Administration's (TSA) processes for developing detection standards--which identify the characteristics of prohibited items, such as explosives, that passenger and checked baggage screening technologies are to detect--among other issues. During the course of our review, we found that TSA had not updated its 2015 guidance for developing new detection standards to reflect key changes in its procedures. Specifically, we found that, as of August 2019, the guidance did not accurately reflect (1) designated procedures for developing detection standards, (2) the roles and responsibilities of key stakeholders, and (3) TSA's organizational structure. For example, the 2015 guidance calls for an annual assessment of emerging threats, which a senior TSA official told us TSA no longer conducts because relevant emerging threats are now occurring more frequently and intelligence information is processed on an ongoing basis. Consequently, we recommended that TSA update the 2015 guidance for developing and approving screening technology explosives detection standards to reflect designated procedures, the roles and responsibilities of stakeholders, and changes in the agency's organizational structure. In November 2019, TSA provided the Requirements Engineering Integrated Process Manual (RE IPM), which presents designated procedures for developing detection standards and specifies the roles and responsibilities of key stakeholders, with references to offices in TSA's organizational structure. Updating its guidance for developing detection standards should provide TSA with better assurance that detection standards are developed in accordance with established policies and practices. As a result, this recommendation is closed as implemented.
|
| Transportation Security Administration | The TSA Administrator should require and ensure that TSA officials document key decisions, including testing and analysis decisions, used to support the development and consideration of new screening technology explosives detection standards. (Recommendation 2) |
Closed – Implemented
In December 2019, we reported on the Transportation Security Administration's (TSA) development of detection standards for passenger and checked baggage screening technologies. During our review of TSA's steps to develop detection standards from fiscal years 2014 through 2018, we found that TSA and DHS's Science and Technology Directorate (S&T) did not document all key decisions--those that could potentially affect outcomes--regarding the testing and analyses (characterization) of explosive threat materials and the development of explosives detection standards. For example, we found that in five of the seven sets of testing and analyses of explosive materials--referred to as material threat assessments--we reviewed, TSA and S&T did not consistently document key steps, such as how selected samples were prepared for testing, and in three assessments officials did not cite standard operating procedures for at least one laboratory. Consequently, we recommended that TSA should require and ensure that TSA officials document key decisions, including testing and analysis decisions, used to support the development and consideration of new screening technology explosives detection standards. In November 2019, TSA provided the Requirements Engineering Integrated Process Manual (RE IPM), which details TSA's process for documenting key decisions, including testing and analysis decisions, used in the development of new screening technology explosives detection standards. In July 2022, TSA provided a new material threat assessment to demonstrate compliance with the 2019 guidance. This assessment clearly documents key testing and analysis decisions as well as how officials prepared selected samples, the methods they used to synthesize material samples for testing, and that standard operating procedures were followed. Documenting key decisions regarding the testing and analyses of explosive threat materials--especially those that could potentially affect outcomes--should help TSA better ensure that effective decisions are made and that organizational knowledge is retained regardless of changes in personnel. As a result, this recommendation is closed as implemented.
|
| Transportation Security Administration | The TSA Administrator should require and ensure that TSA officials document their assessments of risk and the rationale—including the assumptions, methodology, and uncertainty considered—behind decisions to deploy screening technologies. (Recommendation 3) |
Open
In December 2019, we reported that TSA's process for incorporating risk into its plans for deploying screening technologies to specific airports lacks transparency. Officials said their discussions with security and intelligence officials about deployment strategies-including relevant risk information-are generally informal and not documented. Consequently, we recommended that TSA should require and ensure that agency officials document their assessments of risk and the rationale behind decisions to deploy screening technologies. In September 2021, TSA provided the Requirements and Capabilities Analysis (RCA) Deployment Guiding Principles, which is to be used in planning the deployment of screening technologies in, among other things , a risk-informed manner. It specifies that officials should determine the overall risk of airports and use that assessment to inform deployment strategies; it also cites numerous principles officials are to consider in developing a risk-based approach to deployment. While it is commendable that TSA has developed deployment guidance that factors in risk and other principles to be considered or followed when developing deployment plans, the guidance does not require officials to document risk assessments and the rationale behind deployment decisions. In May 2022, TSA provided an action plan and deployment schedule for specific screening technology to selected airports . While the action plan states that deployment decisions should include a "risk based, top down deployment approach when able" as part of the criteria for initial site evaluation and selection, the plan does not explain what, if any risk assessment was performed as part of the site evaluation process, or the rationale for the final deployment schedule. To fully address this recommendation, TSA should require that risk assessments and the rationale be documented as part of screening technology deployment plans. TSA should also provide an example of a screening technology deployment plan that includes a risk assessment and explains the rationale-including the assumptions, methodology, and uncertainty considered-for the deployment of screening technologies. As of August 2023, we are continuing to monitor TSA's implementation of this recommendation.
|
| Transportation Security Administration | The TSA Administrator should develop a process to ensure that screening technologies continue to meet detection requirements after deployment to commercial airports. (Recommendation 4) |
Open – Partially Addressed
In December 2019, we reported that TSA practices do not ensure that screening technologies continue to meet detection requirements after they have been deployed to airports. We recommended that TSA develop a process to ensure that screening technologies continue to meet detection requirements after deployment to commercial airports. In May 2020, TSA provided the Post Implementation and Periodic Review Policy (APM-20-031). According to the Policy, TSA will use Post Implementation Reviews (PIR) to explain how screening technology performance, including detection, is to be assessed over time, following deployment. Each technology system is to require its own PIR-or roadmap for reviewing component performance of the detection chain-because each technology has unique logistics data and detection chain. PIRs are to be conducted within 6 to 12 months after initial operation (or as otherwise designated) and are to determine user satisfaction and system performance relative to effectiveness and suitability, among other things. However, timeframes and other requirements for conducting periodic reviews after the PIR are less clear-for example, system performance requirements "relative to effectiveness and suitability" are not specified for post-PIR (periodic) reviews. In May 2022, TSA stated that the timing of periodic reviews is subject to resource availability and the judgment of the management team for that technology, since each technology requires a unique evaluation and timetable for periodic reviews due to differences in component lifecycles. Similar to the PIR, periodic reviews are to prioritize key performance parameters, which are tied to effectiveness and suitability. To fully develop a process for ensuring screening technologies continue to meet detection requirements after deployment, TSA policy and guidelines should call for the same requirements for periodic reviews as for PIRs, such as general timeframes for conducting reviews that allow for the individual judgement of the management team , and system performance requirements relative to technology effectiveness and suitability. As of August 2023, we are continuing to monitor TSA's implementation of this recommendation.
|
| Transportation Security Administration | The TSA Administrator should implement the process it develops to ensure that screening technologies continue to meet detection requirements after deployment to commercial airports. (Recommendation 5) |
Open
In December 2019, we reported that TSA practices do not ensure that screening technologies continue to meet detection requirements after they have been deployed to airports. In April 2020, TSA issued a policy for developing a review process to assess performance after the deployment of each technology, including detection over time. Since TSA cannot use live explosives or simulants to test screening technologies, the agency plans to measure, for each technology, the performance of system components within the detection chain instead of directly measuring detecting requirements. In 2021, to address this recommendation, TSA provided a report for Explosives Trace Detection (ETD) technology to demonstrate the review process. However, according to TSA, the agency initiated the report prior to the publication of TSA's April 2020 policy for conducting the reviews, and therefore this review does not address all TSA requirements. We informed TSA that its April 2020 policy should require consistent elements across all reviews, such as general timeframes for conducting reviews. To fully address this recommendation, TSA needs to demonstrate that is has developed a review process for the testing of all TSA screening technologies deployed to the field. Since each technology has unique operational parameters and requirements, TSA's plans should address the specific technical approach the agency intends to use to ensure the screening technologies continue to meet detection requirements. As of August 2023, we are continuing to monitor TSA's implementation of this recommendation.
|