Air Cargo Security: TSA Field Testing Should Ensure Screening Systems Meet Detection Standards
Fast Facts
According to the Transportation Security Administration, the threat of explosives in air cargo coming into the United States is significant. TSA requires air carriers to x-ray or screen the cargo by other means before it enters the country.
In 2020, TSA began field testing imaging technology (similar to CT medical imaging) to screen small parcels for explosives. We found TSA didn't follow key design and evaluation practices during testing. As a result, TSA can't ensure that the imaging technology meets detection requirements. We recommended TSA perform additional testing or analysis on the technology before TSA approves it for use.
Highlights
What GAO Found
The Department of Homeland Security's (DHS) Transportation Security Administration (TSA) and U.S. Customs and Border Protection (CBP) address U.S.-bound air cargo security through separate programs and have taken steps to measure their effectiveness. For example, TSA conducts an inspection program to help ensure that air carriers comply with specific cargo-related security requirements, such as requirements related to cargo acceptance, control and custody, and screening procedures.
Air Cargo Pallet and Air Cargo Loaded onto an Aircraft
From January 2020 through April 2021, TSA conducted a field assessment on the use of a computed tomography (CT)-based explosives detection system to screen air cargo as part of its ongoing process to qualify the system for use by air carriers. This type of system produces images of parcels that are examined by computer for signs of explosives. However, TSA's assessment did not fully meet three of five key design and evaluation practices. While the assessment identified goals and established metrics, TSA did not incorporate other key practices, such as collecting all necessary data about the system's ability to detect threats (probability of detection) in the field, consistent with TSA's standards. Since TSA officials cannot use live explosives in the field to measure the probability of detection, they relied on image quality testing, using a manufacturer's test kit to compare system performance in the field with earlier tests performed in a laboratory with live explosives.
However, TSA did not validate that the test kit was an acceptable alternative test method for determining the CT system's probability of detection in the field. TSA did not (1) independently validate that the test kit captures all ways system performance could degrade or (2) collect any of the underlying quantitative data from the test kit. TSA officials told GAO they did not validate the test kit because its performance was certified during laboratory testing at DHS's Transportation Security Laboratory; however, officials from the Transportation Security Laboratory told GAO they do not certify the performance of test kits. Without a suitable alternative testing approach to determine the probability of detection, TSA will not have all relevant data to assess whether the CT system meets TSA's detection standard requirements in the field and should be qualified for use by air carriers.
Why GAO Did This Study
According to DHS—which is responsible for ensuring the security of air cargo transported to the United States—the threat from explosives in air cargo remains significant.
The TSA Modernization Act includes a provision for GAO to review DHS's processes for securing U.S.-bound air cargo and efforts to use CT technology for air cargo screening. This report addresses, among other things, how DHS secures inbound air cargo, and the extent to which TSA's field assessment of a CT screening system included key practices for design and evaluation.
GAO reviewed TSA and CBP air cargo security procedures and documents and analyzed a random sample of air cargo shipment data from calendar year 2019. GAO also interviewed TSA and CBP headquarters and National Targeting Center officials, and interviewed TSA field and air carrier officials regarding operations with two foreign airports, selected based on TSA risk data and the amount of air cargo transported from these airports to the United States. This is a public version of a sensitive report that GAO issued in May 2021. Information that DHS deemed sensitive was omitted.
Recommendations
In the May 2021 report, GAO made four recommendations, including that TSA ensure it collects all necessary data for field assessments of explosives detection systems for air cargo screening prior to qualifying the systems for use by air carriers. DHS agreed with all four recommendations and said TSA and CBP are taking or planning actions to address them.
Recommendations for Executive Action
| Agency Affected | Recommendation | Status Sort descending |
|---|---|---|
| Department of Homeland Security | The Secretary of DHS should ensure that the Administrator of TSA and the Commissioner of CBP establish a documented process to ensure that relevant officials from both agencies are aware of and have access to applicable data to inform their inbound air cargo risk assessment efforts. (Recommendation 1) |
Open
In July 2021, we reported that the Transportation Security Administration's (TSA) and the U.S. Customs and Border Protection (CBP) have separate procedures for assessing risk for inbound air cargo and share information to inform their respective risk assessments; however, they do not have a documented process that ensures the full exchange of relevant air cargo risk data. Consequently, we recommended that the Department of Homeland Security (DHS) should ensure that TSA and CBP establish a documented process to ensure that both agencies are aware of and have access to applicable data to inform inbound air cargo risk assessments efforts. In February 2023, TSA officials informed us they had met with CBP officials in April 2022 and identified CBP air cargo data that may help TSA in its risk assessments. TSA also stated that going forward the two agencies will meet biannually to discuss the sharing of relevant air cargo data. To document this arrangement, TSA and CBP are drafting a Memorandum of Understanding that details the agreed-upon process for sharing relevant air cargo risk assessment data. To fully address this recommendation, TSA and CBP should finalize this Memorandum of Understanding.
|
| Department of Homeland Security | The Administrator of TSA, prior to designating the explosives detection system for air cargo screening currently under evaluation as "qualified" on the air cargo screening technology list, should, to the extent practicable, verify through additional data collection or analysis that the system's probability of detection in the field matches the performance measured in laboratory testing. (Recommendation 2) |
Open
In July 2021, we reported that the Transportation Security Administration (TSA) did not incorporate all key practices in its field assessment on the use of a computed tomography (CT)-based explosives detection system (EDS) to screen air cargo as part of its ongoing process to qualify the system for use by air carriers. For example, we found that TSA officials did not collect all necessary data about the system's ability to detect threats (probability of detection) in the field, consistent with TSA's standards. TSA officials told us they did not measure probability of detection during the field assessment due to the operational difficulties of using live explosives in the field. Rather, they relied on image quality testing, using a manufacturer's test kit, to compare system performance in the field with earlier tests performed in a laboratory with live explosives. However, TSA officials did not validate that the test kit was an acceptable alternative test method for determining the CT system's probability of detection in the field. In their November 2021 update, TSA officials said they had conducted additional analyses to confirm image quality for the air cargo CT system. Moving forward, TSA officials said they plan to address the need for an independent test of image quality with a new test kit they plan to deploy for the checked baggage program. Officials said they plan to use this new test kit to also test the air cargo CT system, rather than relying on the manufacturer-provided kit. We have requested a meeting with TSA officials to more fully discuss these plans and are waiting for TSA's response. To fully address this recommendation, TSA should verify through additional data collection or analysis-such as use of the new test kit-that the CT system's probability of detection in the field matches the performance measured in laboratory testing.
|
| Department of Homeland Security | The Administrator of TSA should ensure that necessary data are collected during field assessments to independently verify that the probability of detection of explosives detection systems for air cargo screening in the field matches the performance measured in laboratory testing, prior to designating systems as "qualified" on the air cargo screening technology list. TSA could provide this verification either through live explosives testing or, when operational considerations limit TSA's ability to use live threat materials, TSA should use an independently validated, fully documented alternative testing strategy. (Recommendation 3) |
Open
In July 2021, we reported that the Transportation Security Administration (TSA) did not select an evaluation approach that included collecting necessary data about the CT system's probability of detection in the field as part of its field assessment. To assess operational effectiveness, TSA guidance states that the test team should gather data on the probability of detection, false alarm rate, and throughput in the operational environment. In November 2022, TSA officials said they were in the process of developing an independently developed and validated image quality test kit to use in daily testing of the agency's passenger checked baggage program rather than relying solely on a manufacturer-provided test kit. Officials said they plan to use this test kit to also test the air cargo CT system. We have requested a meeting with TSA officials to discuss the status of these plans and are waiting for TSA's response. To fully address this recommendation, TSA should demonstrate how use of the new image quality test kit constitutes a viable alternative testing strategy to assess the probability of detection of systems in the field.
|
| Department of Homeland Security | The Administrator of TSA should ensure statistical techniques are used to analyze data from TSA field assessments, including data from the current field assessment, of explosives detection systems for air cargo screening, prior to designating systems as "qualified" on the air cargo screening technology list. This statistical analysis should include the following elements: (1) calculating error values for each quantitative measurement, (2) identifying all necessary performance thresholds, and (3) comparing the measured values and errors against each threshold to determine the statistical confidence of the results. (Recommendation 4) |
Open
In July 2021, we reported that TSA did not conduct the statistical analysis or, in some cases set performance thresholds, necessary to determine whether the computed tomography (CT)-based explosives detection system (EDS) meets requirements. In November 2021, TSA provided an additional field assessment report on the operation of the air cargo CT system. The analysis in this assessment showed a significant increase in the use of statistical methods and is in line with our recommendation, although to fully address the recommendation, TSA should codify these analysis practices for future field assessments. While this November 2021 assessment shows notable improvement in the use of statistical techniques to analyze data, TSA did not apply this type of analysis to every area of its evaluation. In November 2022, TSA stated that it concurred with our recommendation to codify the statistical analysis, acceptance criteria, and performance thresholds and was in the process of developing these data. We have requested a meeting with TSA officials to discuss its progress and are waiting for TSA's response. To fully address this recommendation, TSA should fully analyze data, including (1) calculating error values for each quantitative measurement, (2) identifying all necessary performance thresholds, and (3) comparing measured values and errors against each threshold to determine the statistical confidence of the results. TSA should also codify the use of this statistical analysis for future field assessments.
|