What GAO Found
The Department of Homeland Security (DHS) lacks reliable information about BioWatch Gen-2's technical capabilities to detect a biological attack and therefore lacks the basis for informed cost-benefit decisions about upgrades to the system. DHS commissioned several tests of the technical performance characteristics of the current BioWatch Gen-2 system, but has not developed performance requirements that would enable it to interpret the test results and draw conclusions about the system's ability to detect attacks. Although DHS officials said that the system can detect catastrophic attacks, which they define as attacks large enough to cause 10,000 casualties, they have not specified the performance requirements necessary to reliably meet this operational objective. In the absence of performance requirements, DHS officials said computer modeling and simulation studies support their assertion. However, none of these studies were designed to incorporate test results from the Gen-2 system and comprehensively assess the system against the stated operational objective. Additionally, DHS has not prepared an analysis that combines the modeling and simulation studies with the specific Gen-2 test results to assess the system's capabilities to detect attacks. Finally, we found limitations and uncertainties in the four key tests of the Gen-2 system's performance. Because it is not possible to test the BioWatch system directly by releasing live biothreat agents into the air in operational environments, DHS relied on chamber testing and the use of simulated biothreat agents, which limit the applicability of the results. These limitations underscore the need for a full accounting of statistical and other uncertainties, without which decision makers lack a full understanding of the Gen-2 system's capability to detect attacks of defined types and sizes and cannot make informed decisions about the value of proposed upgrades.
The actions and decisions DHS made regarding the acquisition and testing of a proposed next generation of BioWatch (Gen-3) partially aligned with best practices GAO previously identified for developmental testing of threat detection systems. For example, best practices indicate that resilience testing, or testing for vulnerabilities, can help uncover problems early. While DHS took steps to help build resilience into the Gen-3 testing, future testing could be improved by using more rigorous methods to help predict performance in different operational environments. DHS canceled the Gen-3 acquisition in April 2014, but GAO identified lessons DHS could learn by applying these best practices to the proposed Gen-2 upgrades.
According to experts and practitioners, the polymerase chain reaction (PCR), which detects genetic signatures of biothreat agents, is the most mature technology to use for an autonomous detection system. DHS is considering autonomous detection as an upgrade to Gen-2, because according to DHS, it may provide benefits such as reduction in casualties or clean-up costs. But the extent of these benefits is uncertain because of several assumptions, such as the speed of response after a detection, that are largely outside of DHS's control. As a result, the effectiveness of the responseâ€”and the number of lives that could be savedâ€”is uncertain. Further, an autonomous detection system must address several likely challenges, including minimizing possible false positive readings, meeting sensitivity requirements, and securing information technology networks.
Why GAO Did This Study
DHS's BioWatch program aims to provide early indication of an aerosolized biological weapon attack. Until April 2014, DHS pursued a next-generation autonomous detection technology (Gen-3), which aimed to enable collection and analysis of air samples in less than 6 hours, unlike the current system (Gen-2), which requires manual intervention and can take up to 36 hours to detect the presence of biological pathogens. DHS is taking steps to address the capability gap that resulted from the cancellation of Gen-3 by exploring other technology upgrades and improvements to the Gen-2 system.
GAO was asked to review (1) the technical capabilities of the currently deployed BioWatch system, (2) the Gen-3 testing effort, and (3) characteristics of autonomous detection as a possible option to replace the current BioWatch system. GAO analyzed key program documents, including test plans, test results, and modeling studies. GAO assessed Gen-3 testing against best practices, reviewed relevant literature, and discussed the BioWatch program and testing efforts with key agency officials and national laboratories staff.
GAO recommends DHS not pursue upgrades or enhancements for Gen-2 until it reliably establishes the system's current capabilities. GAO also recommends DHS incorporate best practices for testing in conducting any system upgrades. DHS generally concurred with GAO's recommendations.
Recommendations for Executive Action
|Department of Homeland Security||1. To help ensure that biosurveillance-related funding is directed to programs that can demonstrate their intended capabilities, and to help ensure sufficient information is known about the current Gen-2 system to make informed cost-benefit decisions about possible upgrades and enhancements to the system, the Secretary of Homeland Security should direct the Assistant Secretary for Health Affairs and other relevant officials within the Department to not pursue upgrades or enhancements to the current BioWatch system until the Office of Health Affairs (OHA): (1) establishes technical performance requirements, including limits of detection, necessary for a biodetection system to meet a clearly defined operational objective for the BioWatch program by detecting attacks of defined types and sizes with specified probabilities; (2) assesses the Gen-2 system against these performance requirements to reliably establish its capabilities; and (3) produces a full accounting of statistical and other uncertainties and limitations in what is known about the system's capability to meet its operational objectives.|
|Department of Homeland Security||2. To help reduce the risk of acquiring immature detection technologies, the Secretary of Homeland Security should direct the Assistant Secretary for Health Affairs, in coordination with the Under Secretary for Science and Technology, to use the best practices outlined in this report to inform test and evaluation actions for any future upgrades or changes to technology for BioWatch.|