TSA Acquisitions: TSA Needs to Establish Metrics and Evaluate Third Party Testing Outcomes for Screening Technologies
Fast Facts
The machines that screen airline passengers and baggage must be tested before they can be purchased by the Transportation Security Administration. TSA does most of this testing itself. To make the process more efficient and get new technology into airports faster, TSA allows companies to have testing done by other testers in certain situations.
Companies have used this option just five times since 2013. TSA hopes more companies will take this route in the future, but does not know if the approach makes its process faster or more efficient.
Our 3 recommendations could help TSA determine whether third-party testing is helping it reach its goals.
Highlights
What GAO Found
In 2013, the Transportation Security Administration (TSA) introduced the concept of third party testing—having an independent testing entity verify that a security screening system meets certain requirements. The concept is that screening system vendors would take this additional step either prior to submitting their technologies to TSA or if their system failed TSA's test and evaluation process. The goal is for third party testing to reduce the time and resources that TSA spends on its own testing. However, since introduced, TSA has directed only three vendors that failed TSA tests to use third party testing, with varying outcomes. In two other cases, TSA supplemented its test capabilities by using third party testers to determine that systems installed at airports were working properly. TSA officials and industry representatives pointed to several reasons for third party testing's limited use since 2013, such as the cost to industry to use third party testers and TSA's reluctance to date to accept third party test data as an alternative to its own. Despite this, TSA officials told GAO they hope to use third party testing more in the future. For example, in recent announcements to evaluate and qualify new screening systems, TSA stated that it will require a system that fails TSA testing to go to a third party tester to address the identified issues (see figure).
Example of Use of Third Party Testing When a System Experiences a Failure in TSA's Testing
TSA set a goal in 2013 to increase screening technology testing efficiency. In addition, TSA reported to Congress in January 2020 that third party testing is a part of its efforts to increase supplier diversity and innovation. However, TSA has not established metrics to determine third party testing's contribution toward the goal of increasing efficiency. Further, GAO found no link between third party testing and supplier diversity and innovation. Some TSA officials and industry representatives also questioned third party testing's relevance to these efforts. Without metrics to measure and assess the extent to which third party testing increases testing efficiency, TSA will be unable to determine the value of this concept. Similarly, without assessing whether third party testing contributes to supplier diversity and innovation, TSA cannot know if third party testing activities are contributing to these goals as planned.
Why GAO Did This Study
TSA relies on technologies like imaging systems and explosives detection systems to screen passengers and baggage to prevent prohibited items from getting on board commercial aircraft. As part of its process of acquiring these systems and deploying them to airports, TSA tests the systems to ensure they meet requirements.
The 2018 TSA Modernization Act contained a provision for GAO to review the third party testing program. GAO assessed the extent to which TSA (1) used third party testing, and (2) articulated its goals and developed metrics to measure the effects of third party testing.
GAO reviewed TSA's strategic plans, acquisition guidance, program documentation, and testing policies. GAO interviewed officials from TSA's Test and Evaluation Division and acquisition programs, as well as representatives of vendors producing security screening systems and companies providing third party testing services.
Recommendations
GAO is recommending that TSA develop metrics to measure the effects of third party testing on efficiency, assess its effects on efficiency, and assess whether third party testing contributes to supplier diversity and innovation. DHS concurred with GAO's three recommendations and has actions planned to address them.
Recommendations for Executive Action
| Agency Affected | Recommendation | Status |
|---|---|---|
| Transportation Security Administration | The Administrator of TSA should establish metrics to measure the effects of third party testing on the efficiency of the test and evaluation process. (Recommendation 1) |
Closed – Implemented
TSA concurred with our recommendation. In December 2020, TSA established third party testing metrics, such as the number of retests and test cycle duration, to measure efficiency of the test and evaluation process. As a result, we closed this recommendation as implemented.
|
| Transportation Security Administration | When performance metrics have been established, the Administrator of TSA should assess gains in efficiency resulting from third party testing. (Recommendation 2) |
Open – Partially Addressed
TSA concurred with this recommendation. TSA's Test and Evaluation (T&E) Division assessed security technology qualification testing activities conducted from October 2020 to January 2023 for efficiency gains using the established metrics, the number of retests performed and test cycle duration. As of June 2023, TSA T&E Officials reported that there have been no instances of third party testers used during the assessment period and do not anticipate vendors voluntarily choosing to do so. As such, TSA T&M officials reported no assessed impact of third party testing on the established metrics, and no gains in efficiency resulting from third party testing at this time. GAO will continue to monitor.
|
| Transportation Security Administration | The Administrator of TSA should assess whether third party testing contributes to its goals of increasing supplier diversity and innovation. (Recommendation 3) |
Open – Partially Addressed
TSA concurred with our recommendation. Since October 2020, TSA's Test and Evaluation (T&E) Division continues to conduct qualification testing for security technologies. TSA T&E officials reported that during their assessment period, October 2020 through January 2023, no vendors elected to use third party testers. As of June 2023, TSA T&E officials reported that they found no evidence of third party testing contributing to supplier diversity or innovation goals at this time. GAO will continue to monitor.
|