Aviation Security:

DHS and TSA Have Researched, Developed, and Begun Deploying Passenger Checkpoint Screening Technologies, but Continue to Face Challenges

GAO-10-128: Published: Oct 7, 2009. Publicly Released: Oct 29, 2009.

Additional Materials:

Contact:

Stephen M. Lord
(202) 512-3000
contact@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

Since fiscal year 2002, the Transportation Security Administration (TSA) and the Department of Homeland Security (DHS) have invested over $795 million in technologies to screen passengers at airport checkpoints. The DHS Science and Technology Directorate (S&T) is responsible, with TSA, for researching and developing technologies, and TSA deploys them. GAO was asked to evaluate the extent to which (1) TSA used a risk-based strategy to prioritize technology investments; (2) DHS researched, developed, and deployed new technologies, and why deployment of the explosives trace portal (ETP) was halted; and (3) DHS coordinated research and development efforts with key stakeholders. To address these objectives, GAO analyzed DHS and TSA plans and documents, conducted site visits to research laboratories and nine airports, and interviewed agency officials, airport operators, and technology vendors.

TSA completed a strategic plan to guide research, development, and deployment of passenger checkpoint screening technologies; however, the plan is not risk-based. According to TSA officials, the strategic plan and its underlying strategy for the Passenger Screening Program were developed using risk information, such as threat information. However, the strategic plan and its underlying strategy do not reflect some of the key risk management principles set forth in DHS's National Infrastructure Protection Plan (NIPP), such as conducting a risk assessment based on the three elements of risk--threat, vulnerability, and consequence--and developing a cost-benefit analysis and performance measures. TSA officials stated that, as of September 2009, a draft risk assessment for all of commercial aviation, the Aviation Domain Risk Assessment, was being reviewed internally. However, completion of this risk assessment has been repeatedly delayed, and TSA could not identify the extent to which it will address all three elements of risk. TSA officials also stated that they expect to develop a cost-benefit analysis and establish performance measures, but officials could not provide timeframes for their completion. Without adhering to all key risk management principles as required in the NIPP, TSA lacks assurance that its investments in screening technologies address the highest priority security needs at airport passenger checkpoints. Since TSA's creation, 10 passenger screening technologies have been in various phases of research, development, test and evaluation, procurement, and deployment, but TSA has not deployed any of these technologies to airports nationwide. The ETP, the first new technology deployment initiated by TSA, was halted in June 2006 because of performance problems and high installation costs. Deployment has been initiated for four technologies--the ETP in January 2006, and the advanced technology systems, a cast and prosthesis scanner, and a bottled liquids scanner in 2008. TSA's acquisition guidance and leading commercial firms recommend testing the operational effectiveness and suitability of technologies or products prior to deploying them. However, in the case of the ETP, although TSA tested earlier models, the models ultimately chosen were not operationally tested before they were deployed to ensure they demonstrated effective performance in an operational environment. Without operationally testing technologies prior to deployment, TSA does not have reasonable assurance that technologies will perform as intended. DHS coordinated with stakeholders to research, develop, and deploy checkpoint screening technologies, but coordination challenges remain. Through several mechanisms, DHS is taking steps to strengthen coordination within the department and with airport operators and technology vendors.

Status Legend:

More Info
  • Review Pending-GAO has not yet assessed implementation status.
  • Open-Actions to satisfy the intent of the recommendation have not been taken or are being planned, or actions that partially satisfy the intent of the recommendation have been taken.
  • Closed-implemented-Actions that satisfy the intent of the recommendation have been taken.
  • Closed-not implemented-While the intent of the recommendation has not been satisfied, time or circumstances have rendered the recommendation invalid.
    • Review Pending
    • Open
    • Closed - implemented
    • Closed - not implemented

    Recommendations for Executive Action

    Recommendation: To help ensure that DHS's Science and Technology Directorate (S&T) and Transportation Security Administration (TSA) take a comprehensive, risk-informed approach to the research, development, test and evaluation (RDT&E), procurement, and deployment of airport passenger checkpoint screening technologies, and to increase the likelihood of successful procurements and deployments of such technologies, in the restricted version of this report, we recommended that the Assistant Secretary for TSA should conduct a complete risk assessment, including threat, vulnerability, and consequence assessments, which would apply to the Passenger Screening Program (PSP).

    Agency Affected: Department of Homeland Security: Directorate of Border and Transportation Security: Transportation Security Administration

    Status: Closed - Implemented

    Comments: In October 2009, we reported that the Transportation Security Administration (TSA) had completed a strategic plan to guide research, development, and deployment of passenger checkpoint screening technologies; however, the plan was not risk-based. Specifically, the strategic plan and its underlying strategy for the Passenger Screening Program (PSP) were developed using risk information, such as threat information, but were not based on the three elements of risk--threat, vulnerability, and consequence. We recommended that TSA conduct a complete risk assessment, including threat, vulnerability, and consequence assessments, which would apply to the PSP. TSA concurred and, in response, initiated approaches to identify risks at both the strategic and detailed level. At the strategic level, in March 2010, TSA updated its Aviation Modal Risk Assessment (AMRA), previously called the Aviation Domain Risk Assessment, which includes a comprehensive threat, vulnerability, and consequence assessment of the risk of a terrorist attack on the international and domestic aviation system. TSA has indicated that it will use AMRA to inform its PSP. At a more detailed level, TSA developed a simulation model that can be used to model specific risk scenarios and risk-reduction potential of specific technologies or other countermeasures. According to TSA, the tool will allow the agency to perform analysis at a level not previously available while not disrupting real world checkpoint operations. TSA indicated that the outputs of this tool are being used to inform the strategic planning and research and development processes. This recommendation is closed as implemented.

    Recommendation: To help ensure that DHS's Science and Technology Directorate (S&T) and Transportation Security Administration (TSA) take a comprehensive, risk-informed approach to the RDT&E, procurement, and deployment of airport passenger checkpoint screening technologies, and to increase the likelihood of successful procurements and deployments of such technologies, in the restricted version of this report, we recommended that the Assistant Secretary for TSA should develop cost-benefit analyses to assist in prioritizing investments in new checkpoint screening technologies.

    Agency Affected: Department of Homeland Security: Directorate of Border and Transportation Security: Transportation Security Administration

    Status: Open

    Comments: In October 2009, we reported that the Transportation Security Administration (TSA) had not completed a cost-benefit analysis to prioritize and fund the Passenger Screening Program's (PSP) priorities for investing in checkpoint technologies, as required by the National Infrastructure Protection Plan's (NIPP) risk management framework. At the time of our report, TSA had not developed life-cycle cost estimates of each screening technology the PSP is developing, procuring, or deploying, and could not provide us with information on their priorities for the research and development of checkpoint screening technologies or the processes they followed to develop these priorities. We recommended that TSA develop cost-benefit analyses to assist in prioritizing investments in new checkpoint screening technologies. TSA concurred and completed a life-cycle cost estimate for the PSP in November 2011. While completing a life-cycle cost estimate is an important first step in addressing our recommendation, this recommendation will not be closed until TSA develops cost-benefit analyses to assist in prioritizing investments in new checkpoint screening technologies, as required by the NIPP. Such a cost-benefit analysis is important because it would help decision makers determine which protective measures, for instance, investments in technologies or in other security programs, will provide the greatest mitigation of risk for the resources that are available. In addition, as we reported in June 2012, the November 2011 life-cycle cost estimate does not include an independent cost estimate or an assessment of how changing key assumptions and other factors would affect the estimate. As a result, the document has not been approved by the Department of Homeland Security. TSA has also completed cost-benefit analyses of some of the technologies that are part of the PSP, such as Advanced Imaging Technologies (AIT). However, the agency does not require that cost-benefit analyses be conducted for all technologies in the PSP. It is important, as we reported in October 2009, that TSA conduct cost-benefit analysis of each checkpoint technology that it invests in that weighs the costs and benefits of technologies relative to the costs and benefits of other solutions. Such analysis is important because it helps decision makers determine whether investments in technologies or in other security programs will provide the greatest mitigation of risk for the resources that are available. Without cost-benefit or other analysis to compare the cost and effectiveness of various solutions, the agency cannot determine whether investments in the research and development of new checkpoint technologies or procedures most appropriately mitigate risks with the most cost-effective use of resources. In addition, without knowing the full cost of the technologies that the PSP is developing, procuring, or deploying, TSA could potentially invest in a technology in which the cost outweighs expected benefits. In August 2012, TSA program officials told us that they will work toward completing an overall cost-benefit analysis for PSP technologies and they expect to have it completed and approved by August 2013. Until that effort is complete, this recommendation remains open.

    Recommendation: To help ensure that DHS's Science and Technology Directorate (S&T) and Transportation Security Administration (TSA) take a comprehensive, risk-informed approach to the RDT&E, procurement, and deployment of airport passenger checkpoint screening technologies, and to increase the likelihood of successful procurements and deployments of such technologies, in the restricted version of this report, we recommended that the Assistant Secretary for TSA should develop quantifiable performance measures to assess the extent to which investments in research, development, and deployment of checkpoint screening technologies achieve performance goals for enhancing security at airport passenger checkpoints.

    Agency Affected: Department of Homeland Security: Directorate of Border and Transportation Security: Transportation Security Administration

    Status: Open

    Comments: We reported in October 2009 that the Transportation Security Administration's (TSA) strategy for the Passenger Screening Program (PSP) did not have a mechanism - such as performance measures or other evaluation methods - to monitor, assess, or test the extent to which investments in new checkpoint technologies reduce or mitigate the risk of terrorist attacks. We recommended that TSA develop quantifiable performance measures to assess the extent to which investments in research, development, and deployment of checkpoint screening technologies achieve performance goals for enhancing security at airport passenger checkpoints. TSA concurred and, in March 2011, told us that all major technologies that are intended to be deployed to the checkpoint are required to undergo extensive testing in an operational environment during the Operational Testing and Evaluation (OT&E) phase. In addition, TSA said that it has gained an enhanced capability by opening the TSA Systems Integration Facility (TSIF), which allows TSA to mirror the constraints of an operational environment while reducing the impact on the flying public. Finally, according to TSA, baseline comparisons are performed, when relevant, for the previously installed technologies compared to the proposed upgrades to determine if the increase in abilities is acceptable. In addition to the testing performed prior to deployment, TSA uses several methods to evaluate technologies on an ongoing basis to ensure operational effectiveness. For example, TSA has implemented the Aviation Screening Assessment Program (ASAP) during which detection is measured and data are collected to identify trends. Also, TSA performs covert testing of the vulnerabilities of both the technologies and the processes used at the checkpoint. All of these are beneficial steps. However, TSA has not identified quantifiable measures of progress which would allow the agency to assess the PSP's overall effectiveness, as we recommended. Without measures to monitor the degree to which the TSA's investments in the research, development, and deployment of new screening technologies reduce or mitigate terrorist threats, the agency is limited in its ability to assess the effectiveness of the PSP or the extent to which it complements other layers of security at the checkpoint. As TSA moves forward in developing performance measures, it is important that these measures reflect not only efficiency of the technologies to process passengers but the effectiveness of technologies and other countermeasures to make the checkpoint more secure and, thereby, reduce the risks posed by the most pressing threat scenarios identified in TSA's risk assessments. In August 2012, TSA program officials told us that they would work toward developing overarching, quantifiable performance measures for PSP technologies after the PSP cost-benefit analysis is completed and approved. This recommendation remains open.

    Recommendation: To help ensure that DHS's Science and Technology Directorate (S&T) and Transportation Security Administration (TSA) take a comprehensive, risk-informed approach to the RDT&E, procurement, and deployment of airport passenger checkpoint screening technologies, and to increase the likelihood of successful procurements and deployments of such technologies, in the restricted version of this report, we recommended that the Assistant Secretary for TSA should, after conducting a complete risk assessment and completing cost-benefit analyses and quantifiable performance measures for the PSP, incorporate the results of these efforts into the PSP strategy as determined appropriate.

    Agency Affected: Department of Homeland Security: Directorate of Border and Transportation Security: Transportation Security Administration

    Status: Open

    Comments: We reported in October 2009 that the Transportation Security Administration (TSA) completed a strategic plan to guide research, development, and deployment of passenger checkpoint screening technologies; however, the plan was not risk-based. According to TSA officials, the strategic plan and its underlying strategy for the Passenger Screening Program (PSP) were developed using risk information, such as threat information. However, the strategic plan and its underlying strategy did not reflect some of the key risk management principles set forth in DHS's National Infrastructure Protection Plan (NIPP), such as conducting a risk assessment based on the three elements of risk--threat, vulnerability, and consequence--and developing a cost-benefit analysis and performance measures. Without adhering to all key risk management principles as required in the NIPP, TSA lacked assurance that its investments in screening technologies address the highest priority security needs at airport passenger checkpoints. We recommended that TSA, after conducting a complete risk assessment and completing cost-benefit analyses and quantifiable performance measures for the PSP, incorporate the results of these efforts into the PSP strategy as determined appropriate. TSA concurred and told us in March 2011 that the agency was updating its PSP strategic plan. However, as of August 2012, implementation of our recommendation has been postponed due to TSA's internal reorganization. This recommendation remains open and is contingent upon the implementation of our recommendations on cost-benefit analyses and performance measures and upon the next version of the PSP strategy, to include the results of the efforts called for in those two recommendations.

    Recommendation: To help ensure that DHS's Science and Technology Directorate (S&T) and Transportation Security Administration (TSA) take a comprehensive, risk-informed approach to the RDT&E, procurement, and deployment of airport passenger checkpoint screening technologies, and to increase the likelihood of successful procurements and deployments of such technologies, in the restricted version of this report, we recommended that the Assistant Secretary for TSA should, to the extent feasible, ensure that operational tests and evaluations have been successfully completed before deploying checkpoint screening technologies to airport checkpoints.

    Agency Affected: Department of Homeland Security: Directorate of Border and Transportation Security: Transportation Security Administration

    Status: Closed - Implemented

    Comments: In October 2009, we reported that although the Transportation Security Administration (TSA) had tested earlier models of the explosives trace portal (ETP), the models ultimately chosen were not operationally tested before they were deployed to ensure they demonstrated effective performance in an operational environment. We recommended that TSA, to the extent feasible, ensure that technologies have completed operational tests and evaluations before they are deployed. TSA concurred with the recommendation and has taken actions to address it. For example, TSA implemented a test and evaluation process for all of its technology procurements in accordance with the Department of Homeland Security's policy. This includes developmental testing in the lab and operational testing in the field. Specifically, according to TSA, Operational Test and Evaluation (OT&E) focuses on validating operational effectiveness and suitability. A Test and Evaluation Master Plan (TEMP) is developed to describe the test and evaluation strategy for determining a system's operational effectiveness and suitability. An operational test plan is developed and approved, and an Operational Test Readiness Review is conducted prior to the start of testing. According to TSA, these elements ensure that operational tests and evaluations have been successfully completed before deploying checkpoint screening technologies to airport checkpoints. Subsequently, TSA provided a configuration management plan that, if implemented as required, should ensure that the specific configuration of a checkpoint screening technology that passes OT&E is the same one that is deployed to airports for day-to-day operational use. This recommendation is closed as implemented.

    Recommendation: To help ensure that DHS's Science and Technology Directorate (S&T) and Transportation Security Administration (TSA) take a comprehensive, risk-informed approach to the RDT&E, procurement, and deployment of airport passenger checkpoint screening technologies, and to increase the likelihood of successful procurements and deployments of such technologies, in the restricted version of this report, we recommended that the Assistant Secretary for TSA should evaluate whether TSA's current passenger screening procedures should be revised to require the use of appropriate screening procedures until it is determined that existing emerging technologies meet their functional requirements in an operational environment.

    Agency Affected: Department of Homeland Security: Directorate of Border and Transportation Security: Transportation Security Administration

    Status: Closed - Implemented

    Comments: In October 2009, we reported that the Transportation Security Administration (TSA) had relied on technologies in day-to-day airport operations that had not been demonstrated to meet their functional requirements in an operational environment. For example, TSA substituted existing screening procedures with screening by the Whole Body Imager (later known as Advanced Imaging Technology) even though its performance had not yet been validated by testing in an operational environment. We recommended that TSA evaluate whether current passenger screening procedures should be revised to require the use of appropriate screening procedures until TSA determined that existing emerging technologies meet its functional requirements in an operational environment. TSA agreed with this recommendation. Subsequently, TSA provided us with documentation verifying that the agency had compared the effectiveness of its existing screening procedures and technologies that had not yet completed operational testing and were emerging at the time of our October 2009 report, and considering modifications to those procedures until operational testing had been completed. Therefore, this recommendation is closed as implemented.

    Recommendation: To help ensure that DHS's Science and Technology Directorate (S&T) and Transportation Security Administration (TSA) take a comprehensive, risk-informed approach to the RDT&E, procurement, and deployment of airport passenger checkpoint screening technologies, and to increase the likelihood of successful procurements and deployments of such technologies, in the restricted version of this report, we recommended that the Assistant Secretary for TSA should, in the future, prior to testing or using all checkpoint screening technologies at airports, determine whether TSA's passenger screening procedures should be revised to require the use of appropriate screening procedures until the performance of the technologies has been validated through successful testing and evaluation.

    Agency Affected: Department of Homeland Security: Directorate of Border and Transportation Security: Transportation Security Administration

    Status: Closed - Implemented

    Comments: In October 2009, we reported that the Transportation Security Administration (TSA) had relied on technologies in day-to-day airport operations that had not been demonstrated to meet their functional requirements in an operational environment. For example, TSA substituted existing screening procedures with screening by the Whole Body Imager (now known as Advanced Imaging Technology) even though its performance had not yet been validated by testing in an operational environment. We recommended that TSA should, in the future, prior to testing or using all checkpoint screening technologies at airports, determine whether TSA's passenger screening procedures should be revised to require the use of appropriate screening procedures until the performance of the technologies has been validated through successful testing and evaluation. TSA agreed with this recommendation. In response, TSA revised the Test and Evaluation Master Plan for its Passenger Screening Program to include consideration of how new technologies will integrate with existing screening procedures at the checkpoint and evaluation of the procedures during operational testing. TSA also provided documentation that such consideration had occurred as part of operational testing and evaluation of several new passenger screening technologies. Therefore, this recommendation is closed as implemented.

    Recommendation: To help ensure that DHS's Science and Technology Directorate (S&T) and Transportation Security Administration (TSA) take a comprehensive, risk-informed approach to the RDT&E, procurement, and deployment of airport passenger checkpoint screening technologies, and to increase the likelihood of successful procurements and deployments of such technologies, in the restricted version of this report, we recommended that the Assistant Secretary for TSA should valuate the benefits of the Explosives Trace Portals that are being used in airports, and compare the benefits to the costs to operate and maintain this technology to determine whether it is cost-effective to continue to use the machines in airports.

    Agency Affected: Department of Homeland Security: Directorate of Border and Transportation Security: Transportation Security Administration

    Status: Closed - Implemented

    Comments: In October 2009, we reported that the Transportation Security Administration (TSA) procured and deployed explosives trace portal equipment even though TSA officials were aware that earlier tests did not demonstrate reliable performance in an airport environment. We recommended that TSA conduct an evaluation and determine whether it was cost effective to continue to use these machines. TSA concurred with this recommendation and later halted further deployment of these machines due to performance, maintenance, and installation problems. As of April 2011, TSA reported and provided documented evidence that it had removed from airports all 101 machines that it had deployed. This recommendation is closed as implemented.

    Apr 7, 2014

    Mar 31, 2014

    Mar 28, 2014

    Mar 26, 2014

    Mar 12, 2014

    Mar 7, 2014

    Feb 27, 2014

    Feb 13, 2014

    Looking for more? Browse all our products here