Combating Nuclear Smuggling:

DHS's Phase 3 Test Report on Advanced Portal Monitors Does Not Fully Disclose the Limitations of the Test Results

GAO-08-979: Published: Sep 30, 2008. Publicly Released: Oct 30, 2008.

Additional Materials:

Contact:

Eugene E. Aloise
(202) 512-6870
contact@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

The Department of Homeland Security's (DHS) Domestic Nuclear Detection Office (DNDO) is responsible for addressing the threat of nuclear smuggling. Radiation detection portal monitors are part of the U.S. defense against such threats. In 2007, Congress required that funds for new advanced spectroscopic portal (ASP) monitors could not be spent until the Secretary of DHS certified that these machines represented a significant increase in operational effectiveness over currently deployed portal monitors. In addition to other tests, DNDO conducted the Phase 3 tests on ASPs to identify areas in which the ASPs needed improvement. GAO was asked to assess (1) the degree to which the Phase 3 test report accurately depicts the test results and (2) the appropriateness of using the Phase 3 test results to determine whether ASPs represent a significant improvement over current radiation detection equipment. GAO also agreed to provide its observations on special tests conducted by Sandia National Laboratories (SNL).

Because the limitations of the Phase 3 test results are not appropriately stated in the Phase 3 test report, the report does not accurately depict the results from the tests and could potentially be misleading. In the Phase 3 tests, DNDO performed a limited number of test runs. Because of this, the test results provide little information about the actual performance capabilities of the ASPs. The report often presents each test result as a single value; but considering the limited number of test runs, the results would be more appropriately stated as a range of potential values. For example, the report narrative states in one instance that an ASP could identify a source material during a test 50 percent of the time. However, the narrative does not disclose that, given the limited number of test runs, DNDO can only estimate that the ASP would correctly identify the source from about 15 percent to about 85 percent of the time--a result that lacks the precision implied by DNDO's narrative. DNDO's reporting of the test results in this manner makes them appear more conclusive and precise than they really are. The purpose of the Phase 3 tests was to conduct a limited number of test runs in order to identify areas in which the ASP software needed improvement. While aspects of the Phase 3 report address this purpose, the preponderance of the report goes beyond the test's original purpose and makes comparisons of the performance of the ASPs with one another or with currently deployed portal monitors. In GAO's view, it is not appropriate to use the Phase 3 test report in determining whether the ASPs represent a significant improvement over currently deployed radiation equipment because the limited number of test runs do not support many of the comparisons of ASP performance made in the Phase 3 report. As the report shows, if an ASP can identify a source material every time during a test, but the test is run only five times, the only thing that can be inferred with a high level of statistical confidence is that the probability of identification is no less than about 60 percent. Although DNDO states in the Phase 3 test report that the results will be relevant to the Secretary's certification that the ASPs represent a significant increase in operational effectiveness, it does not clarify in what ways the results will be relevant. Furthermore, DNDO offers no explanation as to why it changed its view from the Phase 3 test plan, which states that these tests will not be used to support a certification decision. The goal of SNL's special tests was, among other things, to identify potential vulnerabilities in the ASPs by using different test scenarios from those that DNDO planned to use in other ASP tests. SNL concluded in its test report that the ASPs' software and hardware can be improved and that rigor could be added to DNDO's testing methods. Furthermore, the report acknowledges that (1) a specific objective of the testing at the Nevada Test Site was to refine and improve the ASP's performance and (2) the special tests were never intended to demonstrate conformity of the ASPs with specific performance requirements. In GAO's view, these statements appear to accurately describe the purpose, limitations, and results of the special tests.

Recommendations for Executive Action

  1. Status: Closed - Implemented

    Comments: The results of and problems with the Phase 3 tests provided "relevant insights into important aspects of system performance" and thus informed the Secretary's decision NOT to certify the ASP as representing a Significant Increase in Operational Effectiveness. The Secretary considered these relevant insights along with the findings of subsequent testing in her decision not to certify ASPs for primary or secondary deployment and ultimately to terminate the program.

    Recommendation: To ensure that the limitations of the Phase 3 test results, and future ASP test results, are clearly understood, and if the Secretary of DHS intends to consider the results of the Phase 3 tests, along with other test data information, in making a certification decision regarding ASPs, then the Secretary should clearly state which "relevant insights into important aspects of system performance" from the Phase 3 report are factored into any decision regarding the certification that ASPs demonstrate a significant increase in operational effectiveness.

    Agency Affected: Department of Homeland Security

  2. Status: Closed - Not Implemented

    Comments: The weaknesses of the Phase 3 test results were ultimately recognized by the Secretary and these results were not used in the certification decision. Instead, these results and the GAO critique of the Phase 3 testing were used to design and implement a much more rigorous round of testing of ASPs in 2008, and subsequent rounds of field testing in the following years. Accordingly, there was no need for the Secretary to order the Director of DNDO to write a report more fully disclosing and articulating the limitations of the Phase 3 tests.

    Recommendation: To ensure that the limitations of the Phase 3 test results, and future ASP test results, are clearly understood, and if the Secretary of DHS intends to consider the results of the Phase 3 tests, along with other test data information, in making a certification decision regarding ASPs, then the Secretary should direct the Director of DNDO to revise and clarify the Phase 3 test report to more fully disclose and articulate the limitations present in the Phase 3 tests--particularly the limitations associated with making comparisons between detection systems from a small number of test runs.

    Agency Affected: Department of Homeland Security

  3. Status: Closed - Implemented

    Comments: Both Secretary Chertoff and Secretary Napolitano ultimately used the Phase 3 test results for the purposes for which they were intended -- to identify areas needing improvement. In part at least because of the problems GAO uncovered with the Phase 3 testing, Congress added a requirement that the Secretary certify that ASPs would deliver a Significant Increase in Operational Effectiveness before moving toward a fullscale procurement and deployment of ASPs. As a result, another, and a much more rigorous round of testing for ASPs were ordered for the summer of 2008 in order to determine this. Accordingly, the Secretary did not use Phase 3 test results for as a justification for certifying ASPs as warranting full scale production. On the contrary, after looking at all the test results for ASPs, including the Phase 3 results, the summer 2008 test results, and three subsequent rounds of field testing, DNDO Director Warren Stern announced the cancellation of the ASP program in testimony before the House Homeland Security Subcommittee on on July 26, 2011. The Secretary followed up with an October 3, 2011 letter to House Homeland Security Committee Chairman Peter King announcing the cancellation of the ASP program.

    Recommendation: To ensure that the limitations of the Phase 3 test results, and future ASP test results, are clearly understood, the Secretary of DHS should use the results of the Phase 3 tests solely for the purposes for which they were intended--to identify areas needing improvement, not as a justification for certifying whether the ASPs warrant full-scale production.

    Agency Affected: Department of Homeland Security

  4. Status: Closed - Implemented

    Comments: DNDO has included information on test limitations in a test report for tests that were conducted after the Phase 3 tests. Specifically, the final March 2009 report on ASP performance testing conducted at the Nevada Test Site has a section on test limitations and constraints which discloses several key limitations. This type of disclosure is consistent with our recommendation.

    Recommendation: Since there are several phases of additional ASP testing currently ongoing, the Secretary should direct the Director of DNDO to take steps to ensure that any limitations associated with ongoing testing are properly disclosed when the results of the current testing are reported.

    Agency Affected: Department of Homeland Security

 

Explore the full database of GAO's Open Recommendations »

Nov 6, 2014

Oct 14, 2014

Sep 30, 2014

Sep 24, 2014

Sep 18, 2014

Sep 17, 2014

Sep 10, 2014

Sep 9, 2014

Looking for more? Browse all our products here