Combating Nuclear Smuggling: DHS's Phase 3 Test Report on Advanced Portal Monitors Does Not Fully Disclose the Limitations of the Test Results
GAO-08-979
Published: Sep 30, 2008. Publicly Released: Oct 30, 2008.
Skip to Highlights
Highlights
The Department of Homeland Security's (DHS) Domestic Nuclear Detection Office (DNDO) is responsible for addressing the threat of nuclear smuggling. Radiation detection portal monitors are part of the U.S. defense against such threats. In 2007, Congress required that funds for new advanced spectroscopic portal (ASP) monitors could not be spent until the Secretary of DHS certified that these machines represented a significant increase in operational effectiveness over currently deployed portal monitors. In addition to other tests, DNDO conducted the Phase 3 tests on ASPs to identify areas in which the ASPs needed improvement. GAO was asked to assess (1) the degree to which the Phase 3 test report accurately depicts the test results and (2) the appropriateness of using the Phase 3 test results to determine whether ASPs represent a significant improvement over current radiation detection equipment. GAO also agreed to provide its observations on special tests conducted by Sandia National Laboratories (SNL).
Recommendations
Recommendations for Executive Action
| Agency Affected | Recommendation | Status |
|---|---|---|
| Department of Homeland Security | To ensure that the limitations of the Phase 3 test results, and future ASP test results, are clearly understood, the Secretary of DHS should use the results of the Phase 3 tests solely for the purposes for which they were intended--to identify areas needing improvement, not as a justification for certifying whether the ASPs warrant full-scale production. |
Closed – Implemented
Both Secretary Chertoff and Secretary Napolitano ultimately used the Phase 3 test results for the purposes for which they were intended -- to identify areas needing improvement. In part at least because of the problems GAO uncovered with the Phase 3 testing, Congress added a requirement that the Secretary certify that ASPs would deliver a Significant Increase in Operational Effectiveness before moving toward a fullscale procurement and deployment of ASPs. As a result, another, and a much more rigorous round of testing for ASPs were ordered for the summer of 2008 in order to determine this. Accordingly, the Secretary did not use Phase 3 test results for as a justification for certifying ASPs as warranting full scale production. On the contrary, after looking at all the test results for ASPs, including the Phase 3 results, the summer 2008 test results, and three subsequent rounds of field testing, DNDO Director Warren Stern announced the cancellation of the ASP program in testimony before the House Homeland Security Subcommittee on on July 26, 2011. The Secretary followed up with an October 3, 2011 letter to House Homeland Security Committee Chairman Peter King announcing the cancellation of the ASP program.
|
| Department of Homeland Security | To ensure that the limitations of the Phase 3 test results, and future ASP test results, are clearly understood, and if the Secretary of DHS intends to consider the results of the Phase 3 tests, along with other test data information, in making a certification decision regarding ASPs, then the Secretary should direct the Director of DNDO to revise and clarify the Phase 3 test report to more fully disclose and articulate the limitations present in the Phase 3 tests--particularly the limitations associated with making comparisons between detection systems from a small number of test runs. |
Closed – Not Implemented
The weaknesses of the Phase 3 test results were ultimately recognized by the Secretary and these results were not used in the certification decision. Instead, these results and the GAO critique of the Phase 3 testing were used to design and implement a much more rigorous round of testing of ASPs in 2008, and subsequent rounds of field testing in the following years. Accordingly, there was no need for the Secretary to order the Director of DNDO to write a report more fully disclosing and articulating the limitations of the Phase 3 tests.
|
| Department of Homeland Security | To ensure that the limitations of the Phase 3 test results, and future ASP test results, are clearly understood, and if the Secretary of DHS intends to consider the results of the Phase 3 tests, along with other test data information, in making a certification decision regarding ASPs, then the Secretary should clearly state which "relevant insights into important aspects of system performance" from the Phase 3 report are factored into any decision regarding the certification that ASPs demonstrate a significant increase in operational effectiveness. |
Closed – Implemented
The results of and problems with the Phase 3 tests provided "relevant insights into important aspects of system performance" and thus informed the Secretary's decision NOT to certify the ASP as representing a Significant Increase in Operational Effectiveness. The Secretary considered these relevant insights along with the findings of subsequent testing in her decision not to certify ASPs for primary or secondary deployment and ultimately to terminate the program.
|
| Department of Homeland Security | Since there are several phases of additional ASP testing currently ongoing, the Secretary should direct the Director of DNDO to take steps to ensure that any limitations associated with ongoing testing are properly disclosed when the results of the current testing are reported. |
Closed – Implemented
DNDO has included information on test limitations in a test report for tests that were conducted after the Phase 3 tests. Specifically, the final March 2009 report on ASP performance testing conducted at the Nevada Test Site has a section on test limitations and constraints which discloses several key limitations. This type of disclosure is consistent with our recommendation.
|
Full Report
Office of Public Affairs
Topics
AccountabilityData integrityDevelopmental testingHomeland securityMaterials researchMonitoringNuclear engineeringNuclear materialsNuclear weaponsNuclear weapons testingOperational testingPerformance measuresProduct evaluationRadiation monitoringReporting requirementsSmugglingSoftwareSystem softwareSystems testingTest equipmentTesting