This is the accessible text file for GAO report number GAO-07-133R 
entitled 'Combating Nuclear Smuggling: DHS's Cost-Benefit Analysis to 
Support the Purchase of New Radiation Detection Portal Monitors Was Not 
Based on Available Performance Data and Did Not Fully Evaluate All the 
Monitors' Costs and Benefits' which was released on October 17, 2006. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as part 
of a longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

United States Government Accountability Office: Washington, DC 20548: 

October 17, 2006: 

The Honorable Thad Cochran:
Chairman:
The Honorable Robert C. Byrd:
Ranking Minority Member:
Committee on Appropriations:
United States Senate: 

The Honorable Jerry Lewis:
Chairman:
The Honorable David R. Obey:
Ranking Minority Member:
Committee on Appropriations:
House of Representatives: 

Subject: Combating Nuclear Smuggling: DHS's Cost-Benefit Analysis to 
Support the Purchase of New Radiation Detection Portal Monitors Was Not 
Based on Available Performance Data and Did Not Fully Evaluate All the 
Monitors' Costs and Benefits: 

Since the attacks of September 11, 2001, combating terrorism has been 
one of the nation's highest priorities. As part of that effort, 
preventing nuclear and radioactive material from being smuggled into 
the United States--perhaps to be used by terrorists in a nuclear weapon 
or in a radiological dispersal device (a "dirty bomb")--has become a 
key national security objective. The Department of Homeland Security 
(DHS) is responsible for providing radiation detection capabilities at 
U.S. ports-of-entry.[Footnote 1] Until April 2005, U.S. Customs and 
Border Protection (CBP) under DHS managed this program. However, on 
April 15, 2005, the president directed the establishment, within DHS, 
of the Domestic Nuclear Detection Office (DNDO), whose duties include 
acquiring and supporting the deployment of radiation detection 
equipment.[Footnote 2] CBP continues its traditional screening function 
at ports-of-entry to interdict dangerous nuclear and radiological 
materials through the use of radiation detection equipment, including 
portal monitors. The Pacific Northwest National Laboratory (PNNL), one 
of the Department of Energy's (DOE) national laboratories, manages the 
deployment of radiation portal monitors for DHS[Footnote 3]. 

Current portal monitors, which cost about $55,000 per monitor, detect 
the presence of radiation, but cannot distinguish between harmless 
radiological materials, such as naturally occurring radiological 
material in some ceramic tile, and dangerous nuclear materials, such as 
highly enriched uranium (HEU). CBP officers also use radioactive 
isotope identification devices (RIIDs), which are handheld devices 
designed to identity different types of radioactive material, such as 
radioactive material used in medicine or industry, a naturally 
occurring source of radiation, or weapons-grade material. These devices 
have limitations in their ability to detect and identify nuclear 
material. DHS would like to improve the capabilities of its radiation 
detection equipment in order to better distinguish between different 
types of nuclear and radiological materials. As a result, DHS sponsored 
research, development, and testing activities in 2005 that were 
designed to produce portal monitors that, in addition to detecting, 
would also identify the type of nuclear or radiological material. 
Portal monitors with this new identification technology currently cost 
about $377,000 or more per monitor. In these same tests, DHS also 
tested the performance of currently deployed portal monitors. 

In July 2006, DHS announced that it had awarded contracts to three 
vendors to further develop and purchase $1.2 billion worth of new 
portal monitors over 5 years. DHS plans to deploy these monitors at 
U.S. ports of entry. For fiscal year 2007, DNDO plans to acquire the 
first installment of 104 new portal monitors that use new 
identification technology at a cost of $80.2 million. Congress, 
however, has curtailed DNDO's ability to do so by restricting the 
availability of funding for full scale procurement of new radiation 
detection portal monitors until DHS certifies that a significant 
increase in operational effectiveness will be achieved.[Footnote 4] 

In a March 2006 report, to ensure that DHS's substantial investment in 
radiation detection technology yields the greatest possible level of 
detection capability at the lowest possible cost, we recommended that 
once the costs and capabilities of these new portal monitors were well 
understood, and before any of the new equipment was purchased, the 
Secretary of DHS work with the Director of DNDO to analyze the benefits 
and costs of deploying new portal monitors.[Footnote 5] Further, we 
recommended that this analysis focus on determining whether any 
additional detection capability provided by the new portal monitors 
were worth the additional costs. 

In response to our recommendation, in May 2006, DNDO issued a cost- 
benefit analysis for the acquisition and deployment of new portal 
monitors.[Footnote 6] In this document, DNDO stated that the purpose of 
its analysis was to help provide a "robust" defense against nuclear 
smuggling, to limit the negative impacts to legitimate trade and travel 
between nations, and to provide a sound financial investment for the 
United States government. In this context, by agreement with your 
staff, we reviewed DNDO's cost-benefit analysis to determine the extent 
to which it provides a sound analytical basis for acquiring and 
deploying new portal monitors. We briefed your staff on the results of 
our evaluation on August 21, 2006, and this report presents the details 
of that briefing. The briefing focused mainly on the use of performance 
and cost data for current and newer portal monitors (see Enclosure 1). 

To conduct our review of DNDO's cost-benefit analysis we analyzed 
documentation supporting the costs used for both currently deployed and 
new equipment. As provided by federal regulations, DHS developed what 
is called an Independent Government Cost Estimate.[Footnote 7] We 
obtained and evaluated this estimate to determine its reasonableness. 
We also reviewed test reports for new portal monitors and currently 
deployed monitors produced by National Institutes of Standards and 
Technology (NIST) and PNNL to assess the validity of performance 
assumptions DNDO used in its analysis. In addition, we had discussions 
with officials from DNDO, CBP, NIST, and several of DOE's national 
laboratories, including Oak Ridge, Sandia, Brookhaven, and PNNL, as 
well as representatives of the trucking and shipping industries. We 
conducted our review from May 2006 to September 2006 in accordance with 
generally accepted government auditing standards. 

Summary: 

DNDO's cost-benefit analysis does not provide a sound analytical basis 
for DNDO's decision to purchase and deploy new portal monitor 
technology. DNDO did not use the results of its own performance tests 
in its cost-benefit analysis and instead relied on assumptions of the 
new technology's anticipated performance level. Performance tests also 
showed that the ability of new radiation detection portal monitors to 
correctly identify masked HEU (placed next to or within another, 
usually more benign, radiological substance) was even more 
limited.[Footnote 8] According to the cost-benefit analysis and 
radiation detection experts to whom we spoke, masked HEU is a 
significant concern because it is difficult to detect. DNDO also 
focused the analysis exclusively on identifying HEU and did not 
consider in the analysis how well (either as a goal or in testing) new 
portal monitor technology can correctly detect or identify other 
dangerous radiological or nuclear materials. Furthermore, the analysis 
did not include the results from side-by-side tests that DNDO conducted 
of the advanced portal monitors and current portal monitors. 

The cost-benefit analysis for acquiring and deploying portal monitors 
is also incomplete because it does not include all of the major costs 
and benefits required by DHS guidelines. In particular, DNDO did not 
assess the likelihood that radiation detection equipment would either 
misidentify or fail to detect nuclear or radiological material. Rather, 
it focused its analysis on reducing the time necessary to screen 
traffic at border check points and reduce the impact of any delays on 
commerce. DNDO also used questionable assumptions about the procurement 
costs of portal monitor technology. DNDO assumed a purchase price for 
current portal monitor technology that is more than twice what CBP 
typically pays. 

We provided DHS with a draft of this report for its review and comment. 
Its written comments are presented in Enclosure 2. The Department 
neither agreed nor disagreed with our recommendation, but continues to 
stand behind the basic conclusions of its cost-benefit analysis that 
new radiation detection portal monitors are a sound investment for the 
U.S. Government. We continue to believe, however, that significant 
weaknesses in DNDO's cost-benefit analysis precludes it from providing 
a reliable basis for making major procurement decisions, such as 
whether to invest heavily in deploying a new portal monitor technology. 

Background: 

DHS's program to deploy radiation detection equipment at U.S. ports-of- 
entry has two goals.[Footnote 9] The first is to use this equipment to 
screen all cargo, vehicles, and individuals coming into the United 
States. Each day, DHS processes about 64,000 containers arriving in the 
United States via ships, trucks, and rail cars; 365,000 vehicles; and 
more than 1.1 million people. The United States has over 380 border 
sites at which DHS plans to deploy radiation detection equipment. The 
second goal of the program is to screen all of this traffic without 
delaying its movement into the nation. To illustrate the difficulty of 
achieving this second goal, CBP's port director at the San Ysidro, 
California, land border crossing estimated that, prior to initiating 
radiation detection screening, the volume of traffic through the port- 
of-entry was so great that, at times, the wait to enter the United 
States from Mexico was about 2.5 hours. He noted that had radiation 
detection screening added a mere 20 seconds to the wait of each 
vehicle, the wait during those peak times could have increased to about 
3.5 or 4 hours--an unacceptable outcome in his view. DHS's current 
plans call for completing deployments of radiation portal monitors at 
U.S. ports-of-entry by September 2009. 

To screen commerce for radiation, CBP uses several types of detection 
equipment and a system of standard operating procedures. Current 
radiation detection equipment includes radiation portal monitors, which 
can detect gamma radiation (emitted by radiological materials that can 
be used in a dirty bomb as well as naturally occurring radiological 
material in some ceramic tile) and neutron radiation (emitted by only a 
limited number of materials, such as plutonium--a material that can be 
used to make a nuclear weapon). 

Generally, CBP's standard procedures direct vehicles, containers, and 
people coming into the country to pass through portal monitors to 
screen for the presence of radiation. This "primary inspection" serves 
to alert CBP officers that a radioactive threat might be present. 
Because current portal monitors using detection technology--known as 
"plastic scintillators" (PVT)--detect the presence of radiation but 
cannot distinguish between harmless and dangerous nuclear or 
radiological materials, "secondary inspections" are required. To 
confirm and identify the presence of radiation, this secondary 
inspection includes CBP officers using RIIDs to determine whether the 
radiation being emitted is from a harmless source, such as kitty 
litter, or a dangerous source, such as weapons-grade nuclear material. 
Typically, completing a secondary inspection takes about 15 minutes. 
DHS would like to improve the capabilities of its radiation detection 
equipment in order to distinguish among different types of nuclear and 
radiological materials and, in so doing, reduce the number of secondary 
inspections on non-threatening nuclear and radiological 
material.[Footnote 10] 

DNDO Relied on the Potential Performance of New Portal Monitors Rather 
Than Its Own Test Results of Current Equipment Performance in Its Cost- 
Benefit Analysis: 

DHS is developing new portal monitors, known as "advanced spectroscopic 
portal monitors" (ASP) that, in addition to detecting nuclear or 
radiological material, can also identify the type of material. In 2005, 
DNDO conducted side-by-side testing at the Nevada Test Site 
(NTS)[Footnote 11] on 10 ASP systems and 3 PVT systems developed by 
private sector companies, including the PVT systems currently deployed. 
DHS requested that NIST provide assistance by conducting an independent 
analysis of data acquired during the last phase of developmental 
testing of ASPs to help DHS determine the performance of ASP portal 
monitors being proposed by private sector companies. NIST compared the 
10 ASP systems, and in June 2006 submitted a report to DHS on the 
results of that testing.[Footnote 12] 

Performance tests of ASPs showed that they did not meet DNDO's main 
performance assumption in the cost-benefit analysis of correctly 
identifying HEU 95 percent of the time it passes through portal 
monitors. The 95 percent performance assumption included ASPs' ability 
to both detect bare, or unmasked, HEU in a container and HEU masked in 
a container with a more benign radiological material.[Footnote 13] 
Based on NIST's assessment of the performance data, the ASP prototypes 
(manufactured by the three companies that won DNDO's recent ASP 
procurement contract) tested at NTS identified bare HEU only 70 to 88 
percent of time. Performance tests also showed that ASPs' ability to 
identify masked HEU fell far short of meeting the 95 percent level of 
performance assumed in the cost-benefit analysis. According to DNDO's 
cost-benefit analysis, identifying masked HEU is the most difficult 
case to address. DOE officials told us that benign radiological 
materials could be used to hide the presence of HEU. NIST reported that 
the best ASP prototype DNDO tested last year in Nevada that won a 
procurement contract was able to correctly identify masked HEU and 
depleted uranium (DU) only 53 percent of the time. Similarly, the ASP 
prototypes submitted by the other two companies that won DNDO's ASP 
procurement were able to identify masked HEU and DU only 45 percent and 
17 percent of the time. 

Despite these results, DNDO did not use the information from these 
tests in its cost-benefit analysis. Instead, DNDO officials told us 
that since new portal monitors cannot meet the 95 percent level of 
performance, they relied on the assumption that they will reach that 
level of performance sometime in the future. DNDO officials asserted 
that the current performance levels of the ASPs will improve, but they 
provided no additional information as to how the 95 percent performance 
goal will be achieved or an estimate of when the technology will attain 
this level. 

Moreover, DNDO's cost-benefit analysis only considered the benefits of 
ASPs' ability to detect and identify HEU and did not consider ASPs 
ability to detect and identify other nuclear and radiological 
materials. The ability of an ASP to identify specific nuclear or 
radiological materials depends on whether the ASP contains software 
that is specific to those materials. In our view, a complete cost- 
benefit analysis would include an assessment of ASPs' ability to detect 
and identify a variety of nuclear and radiological material, not just 
HEU. By excluding radiological and nuclear materials other than HEU, 
DNDO's analysis did not consider the number of secondary inspections 
that would be related to these materials and hence it likely 
underestimated the costs of ASP use. Further, DNDO told us the 
assumptions for the ability of ASP systems to detect and identify HEU 
95 percent came from the ASP performance specifications. However, we 
examined the performance specifications and found no specific 
requirement for detecting or identifying HEU with a 95 percent 
probability.[Footnote 14] While there is a requirement in the 
performance specification for the identification for HEU and other 
special nuclear material, we found no associated probability of success 
in performing this function. 

DNDO's cost-benefit analysis also may not accurately reflect the 
capabilities of PVTs to detect nuclear or radiological material. DNDO 
officials acknowledged that DNDO tested the performance of PVTs along 
with the ASPs in 2005, but did not use the results of these tests in 
its cost-benefit analysis. Instead, in a June 2006 written response to 
our questions, DNDO told us it based its analysis on the performance of 
PVT monitors that PNNL tested during 2004 in New York. However, the 
results from these field tests are not definitive because, as PNNL 
noted in its final report, the tests did not use HEU and, therefore, 
the results from the tests did not indicate how well PVT portal 
monitors would be able to detect HEU in the field.[Footnote 15] 
Moreover, the PVT portal monitors that PNNL used had only one radiation 
detection panel as opposed to the four-panel PVT monitors that DHS 
currently deploys at U.S. ports-of-entry. An expert at a national 
laboratory told us that larger surface areas are more likely to detect 
radiological or nuclear material. DNDO also stated that due to the 
nature of the testing at the Nevada Test Site, the tests did not 
provide the data needed for the cost-benefit analysis. According to 
DNDO officials, this data would come from analysis of the performance 
of fielded systems at U.S. ports-of-entry where the probability to 
detect threats could be compared to false alarm rates. 

DNDO's Cost-Benefit Analysis was Incomplete and Used Inflated Cost 
Estimates for PVT Equipment: 

DNDO officials told us they did not follow the DHS guidelines for 
performing cost-benefit analyses in conducting their own cost-benefit 
analysis.[Footnote 16] DNDO officials said their analysis was intended 
to be an easy-to-understand assessment of the best investment for the 
U.S. government. However, DNDO's simplified approach to the cost- 
benefit analysis omitted many factors that could affect the cost of new 
radiation portal monitors. For example, DNDO officials told us that 
there are over 12 different types of ASP monitors, yet they only 
estimated the cost of cargo portal monitors that would be used at land 
border crossings. In reality, DNDO and CBP plan to deploy different 
types of ASPs that would have varying costs, such as portal monitors at 
seaports which would have higher costs. Additionally, DNDO did not 
capture all the costs related to developing the different types of ASP 
monitors. 

DNDO has also underestimated the life-cycle costs for operations and 
maintenance for both PVT and ASP equipment over time. DNDO analysis 
assumed a 5-year life-cycle for both PVT and ASP equipment. However, 
DNDO officials told us that a 10-year life-cycle was a more reasonable 
expectation for PVT and ASP equipment. The analysis assumes that the 
annual maintenance costs for PVT and ASP monitors will each equal 10 
percent of their respective procurement costs. This means that 
maintenance costs for PVTs would be about $5,500 per year per unit 
based on a $55,000 purchase price and ASP maintenance costs would be 
about $38,000 per year per unit based on a $377,000 purchase price. 
With the much higher maintenance costs for ASPs and doubling the life- 
cycle to 10 years, the long-term implications for these cost 
differences would be magnified. As a result of this, DNDO's analysis 
has not accounted for about $181 million in potential maintenance costs 
for ASPs alone. 

Furthermore, DNDO did not assess the likelihood that radiation 
detection equipment would either misidentify or fail to detect nuclear 
and radiological materials. Rather, DNDO's cost-benefit analysis 
focuses on the ability of ASPs to reduce false alarms--alarms that 
indicate nuclear or radiological material is in a container when, in 
fact, the material is actually non-threatening, such as ceramic tile. 
Reducing false alarms would reduce the number of secondary inspections 
of non-threatening nuclear and radiological materials and therefore the 
costs of those inspections. However, as required by DHS's guide to 
performing cost-benefit analyses, DNDO's analysis does not include all 
costs. In particular, the analysis does not include the potentially 
much bigger cost of "false negatives." False negatives are instances in 
which a container possesses a threatening nuclear or radiological 
material, but the portal monitor either misidentifies the material as 
non-threatening or does not detect the material at all, thus allowing 
the material to enter the country. During the 2005 Nevada tests, the 
incidence of false negatives among the three vendors who received 
contracts ranged from about 45 percent to slightly more that 80 
percent. This raises concerns because, as explained to us by a 
scientist at a national laboratory, at this level of performance, ASPs 
could conceivably misidentify HEU as a benign nuclear or radiological 
material or not detect it at all, particularly if the HEU is placed 
side by side with a non-threatening material such as kitty litter. 

In addition, in recent testimonies before Congress, DNDO's Director has 
cited the primary benefit of deploying ASP monitors as reducing 
unnecessary secondary inspections.[Footnote 17] However, DNDO's 
analysis does not fully estimate today's baseline costs for secondary 
inspections, which makes it impossible to determine whether the use of 
ASPs, as currently planned, will result in significant cost savings for 
these inspections. Finally, the ASP contract award has exceeded DNDO's 
estimate for total cost by about $200 million. The cost-benefit 
analysis shows the total cost for deploying PVT and ASP monitors to be 
about $1 billion, which covers all costs related to acquisition (for 
both PVT and ASP), design, maintenance and physical inspection over 5 
years. However, in July 2006, DHS announced that it had awarded 
contracts to develop and purchase up to $1.2 billion worth of ASP 
portal monitors over 5 years. Furthermore, the cost-benefit analysis 
underestimates total deployment costs and does not account for other 
major costs, such as physical inspections of cargo containers, an 
additional procurement of 442 new PVT monitors, installation and 
integration, and maintenance. 

DNDO's Cost-Benefit Analysis Used Inflated Cost Estimates for PVT 
Equipment: 

DNDO overstated the purchase price of PVT monitors. Although DHS is 
currently paying an average of about $55,000 per monitor, DNDO's cost- 
benefit analysis assumed the PVT would cost $130,959--the highest 
published manufacturers' price for the government.[Footnote 18] 
According to DNDO's Director, DNDO chose the highest published price 
because the current contract for portal monitors is going to expire, 
and the portal monitors will probably cost more in the future. However, 
the information DNDO provided us does not explain why DNDO assumes that 
this future price will be more than double what DHS is currently 
paying, as assumed in DNDO's analysis. PNNL officials told us that the 
future price will almost certainly be lower than the price used in 
DNDO's analysis. 

Conclusions: 

DNDO's cost-benefit analysis does not justify its recent decision to 
spend $1.2 billion to purchase and deploy ASP technology. In 
particular, DNDO used incomplete and unreliable data to evaluate the 
costs and benefits of ASPs. For example, DNDO did not use its own test 
results that showed that the new portal monitor technology could not 
meet the cost-benefit analysis' main performance assumption of 
correctly identifying HEU 95 percent of the time. In addition, the 
ability of ASPs to correctly identify masked HEU falls far short of the 
95 percent level of performance. Instead of relying on performance 
data, DNDO relied on potential future performance to justify the 
purchase of ASPs. While DNDO officials asserted that the current 
performance levels of the ASPs will improve, they provided no 
additional information as to how or when the 95 percent performance 
goal will be achieved. Furthermore, DNDO's simplified approach to the 
cost-benefit analysis omitted many factors that could affect the cost 
of new radiation portal monitors, such as underestimating the life- 
cycle costs for operating and maintaining the equipment over time by 
about $181 million. For these reasons, DNDO's cost-benefit analysis 
does not meet the intent of our March 2006 report recommendation to 
fully assess the costs and benefits before purchasing any new 
equipment. 

Recommendations for Executive Action: 

We recommend that DHS and DNDO conduct a new cost-benefit analysis to 
justify the purchase of new radiation detection portal monitors by 
using sound analytical methods, including actual performance data and a 
complete accounting of all major costs and benefits as required by DHS 
guidelines. We also recommend that DNDO conduct realistic testing for 
both ASPs and PVTs in order to meet the Congressional requirement 
outlined in the Homeland Security Appropriations Act for Fiscal Year 
2007 that DHS certify that a "significant increase in operational 
effectiveness" will be achieved by ASPs before spending additional 
funds to purchase ASPs for deployment. 

Agency Comments and Our Evaluation: 

We provided DHS with a draft of this report for its review and comment. 
Its written comments are presented in Enclosure 2. The Department 
neither agreed nor disagreed with our recommendation, but continues to 
stand behind the basic conclusions of its cost-benefit analysis that 
the ASP program is a sound investment for the U.S. Government. In our 
view, the cost-benefit analysis cannot be used as a reliable basis for 
making a major procurement decision. 

In its written comments, the Department stated that there are three 
areas of disagreement and/or misunderstanding with GAO. In our view, 
however, there are no misunderstandings on any of these issues. First, 
although the Department agreed that performance tests of ASPs did not 
meet DNDO's main performance assumption of correctly identifying HEU 95 
percent of the time it passes through the portal monitors, these tests 
were designed to determine which contractors would manufacture the 
equipment, not determine the absolute capabilities of ASPs. However, in 
our view, an objective and complete cost-benefit analysis should have 
included empirical data on how well ASPs are currently performing. 
Instead of relying on performance data, DNDO relied on potential future 
performance to justify the purchase of ASPs. 

Second, according to the Department, DNDO did not use performance data 
of PVTs in its cost-benefit analysis from tests conducted at the Nevada 
Test Site in 2005 because those tests were not designed to provide an 
objective side-by-side comparison of ASPs and PVTs. However, the 
purpose of the cost-benefit analysis was to determine the benefits, if 
any, of using ASPs instead of currently deployed radiation detection 
technology, including PVTs. The relative capabilities of the two 
competing detection systems--ASPs and PVTs--should have been critical 
to DNDO's analysis. However, DNDO missed an opportunity to collect 
objective, empirical data on the relative capabilities of ASPs and PVTs 
because the agency failed to design its side-by-side tests to measure 
such differences. As a result, instead of having empirical data 
available for its cost-benefit analysis, DNDO relied on assumptions of 
future ASP performance and on limited, unreliable data drawn from tests 
conducted in 2004 to assess the performance of PVTs. DNDO will not be 
able to assess objectively whether ASPs offer any advantages over PVTs, 
much less whether any such potential advantages are worth the ASPs' 
considerable additional cost, until DNDO conclusively determines the 
relative effectiveness of the two technologies. 

Lastly, while the Department agreed that radiation detection portal 
monitors should also be able to detect and identify radiological 
materials in addition to HEU, it believes that HEU is the most 
difficult material to detect and it will continue use HEU as a 
reasonable threat baseline. We agree that HEU is the most difficult 
material to detect, but we disagree that HEU should be used as the only 
reasonable threat baseline to determine the performance of portal 
monitors. As we reported, a complete cost-benefit analysis would 
include an assessment of ASPs' ability to detect and identify a variety 
of nuclear and radiological material, not just HEU. By excluding 
radiological and nuclear materials other than HEU, DNDO's analysis did 
not consider the number of secondary inspections that would be related 
to these materials and hence it likely underestimated the costs of ASP 
use. 

We are sending copies of this correspondence to interested 
congressional committees and members, the Secretary of Homeland 
Security, and other interested parties. We will also make copies 
available to others on request. In addition, this correspondence will 
be available at no charge on GAO's Web site at [Hyperlink, 
http://www.gao.gov]. Should you or your staff have any questions, 
please contact me at (202) 512-6870 or by e-mail at aloisee@gao.gov. 
Contact points for our Office of Congressional Relations and Public 
Affairs may be found on the last page of this correspondence. Key 
contributions to this report include Leland Cogliani, Nancy Crothers, 
Jonathan Fremont, Jim Shafer, Daren Sweeney, and Eugene Wisnoski. 

Signed by: 

Gene Aloise: 
Director, Natural Resources and Environment: 

Enclosures - 2: 

[End of Section] 

Enclosure I: Briefing to House and Senate Committees on Appropriations, 
August 21, 2006: 

Briefing to House and Senate Committees on Appropriations: 

August 21, 2006: 

Analysis of the Domestic Nuclear Detection Office/Department of 
Homeland Security's Cost Benefit Analysis for Next Generation Radiation 
Detection Equipment: 

Introduction: 

The Domestic Nuclear Detection Office (DNDO), within the Department of 
Homeland Security (DHS), is responsible for acquiring and supporting 
the deployment of radiation detection equipment at U.S. ports-of-entry. 

The purpose of the radiation detection equipment is to screen all 
cargo, vehicles, and individuals coming into the United States for 
radiological and nuclear materials, without delaying movement of this 
traffic into the nation. 

Current plastic portal monitors (PVT) are able to detect the presence 
of radiation. DNDO has sponsored research, development, and testing 
activities designed to produce portal monitors that can detect 
radiation and identify the type of material. These new portal monitors 
are known as "advanced spectroscopic portals" (ASP). 

In our March 2006 report[Footnote 19], we recommended that once the 
costs and capabilities of ASPs were well understood, and before any of 
the new equipment was purchased, the Secretary of DHS work with the 
Director of DNDO to analyze the benefits and costs of deploying ASPs. 

We further recommended that this analysis focus on determining whether 
any additional detection capability provided by the ASPs are worth the 
additional costs. 

In response to GAO's recommendation, in May 2006, DNDO issued a cost- 
benefit analysis to help guide its strategy for the acquisition and 
deployment of portal monitors. 

DNDO's Cost-Benefit Analysis: Purpose and Goals: 

The purpose of the analysis was to identify the best strategy that the 
DNDO-Customs and Border Protection could use to meet their deployment 
requirements for nuclear and radiological detection equipment at U.S. 
ports of entry. 

The analysis had three goals: to (1) provide a robust defense against 
nuclear and radiological threats, (2) limit impacts to commerce, and 
(3) provide a sound financial investment for the U.S government. 

Objective: 

The House and Senate Appropriations Committees asked us to review 
DNDO's cost-benefit analysis to determine the extent to which it 
provides a sound analytical basis for acquiring and deploying portal 
monitors. 

Scope and Methodology: 

To conduct our review of DNDO's cost -benefit analysis, we: 

* Analyzed documentation supporting the costs used for both currently 
deployed and new equipment, such as the Independent Government Cost 
Estimate for ASPs. 

* Reviewed government test reports for ASP and PVT systems to assess 
the validity of assumptions DNDO used in its analysis. 

* Had discussions with officials from DNDO, Customs and Border 
Protection, Department of Energy laboratories, National Institutes of 
Standards and Technology, and representatives of the trucking and 
shipping industries. 

DNDO Relied on the Potential Performance of New Portal Monitors Rather 
Than Its Own Test Results in Its Cost-Benefit Analysis: 

DNDO assumed in its cost-benefit analysis that ASPs can correctly 
detect and identify highly enriched uranium (HEU) 95 percent of the 
time it passes through portal monitors. 

In 2005, DNDO tested ASP capabilities to identify both bare, or 
unmasked, HEU and HEU masked in a container with radiological material, 
such as kitty litter. 

Performance tests showed that ASPs identified HEU much less than 95 
percent of the time. At best, the tested ASPs correctly identified 
masked HEU only about half of the time. 

DNDO did not use the results of these tests in its cost-benefit 
analysis and instead assumed that the ASPs will reach that level of 
performance in the future. 

DNDO's Analysis May Not Accurately Reflect the Capabilities of PVTs to 
Detect Radiological Materials: 

DNDO officials acknowledged that DNDO tested the performance of PVTs 
along with the ASPs in 2005, but did not use the results of these tests 
in its cost-benefit analysis. 

Instead, DNDO based its analysis on the performance of PVT monitors 
that PNNL tested during 2004 in New York.  

The PVT monitors that DHS used in the tests had only one radiation 
detection panel as opposed to the four-panel PVT monitors that DHS 
currently deploys at ports-of-entry. 

Additionally, PNNL noted that the results from the tests did not 
indicate how well PVT portal monitors would be able to detect HEU in 
the field. 

DNDO's Cost-Benefit Analysis was Incomplete: 

DNDO officials told us they did not follow DHS guidelines for 
performing cost-benefit analyses. 

DNDO's cost-benefit analysis only considered the benefits of ASPs 
ability to detect and identify HEU and did not consider ASPs ability to 
detect and identify other nuclear and radiological materials. 

A complete analysis would include an assessment of an ASPs ability to 
identify a variety of radiological and nuclear sources, not just HEU. 

DNDO's analysis does not include the bigger cost of instances in which 
a container possesses a threatening radiological or nuclear material, 
but the portal monitor either misidentifies the material as non- 
threatening or does not detect the material at all, thus allowing the 
material to enter the country. 

DNDO analysis assumed a 5-year life-cycle for both PVT and ASP 
equipment. However, DNDO officials told us that a 10-year life-cycle 
was a more reasonable expectation for PVT and ASP equipment. 

Maintenance costs for PVTs would be about $5,500 per year per unit 
based on a $55,000 purchase price and ASP maintenance costs would be 
about $38,000 per year per unit based on a $377,000 purchase price. 

With the much higher maintenance costs for ASPs and doubling the life- 
cycle to 10 years, the long-term implications for these cost 
differences would be magnified. As a result of this, DNDO's analysis 
has not accounted for about $181 million in maintenance costs for ASPs 
alone. 

DNDO's Used Inflated Cost Estimates for PVT Equipment: 

Incalculating costs and benefits, DNDO overstated the acquisition costs 
of PVTs. 

DNDO assumed PVTs would cost about $131,000 per unit-the highest 
published manufacturers' price for the government. 

However, DHS's actual procurement cost for a PVT monitor currently 
averages about $55,000. 

DNDO's Analysis Clearly Underestimates Total Deployment Costs: 

DNDO's analysis shows the total cost for deploying PVT and ASP monitors 
to be about $1 billion; which covers all costs related to acquisition 
(for both PVT and ASP), design, maintenance and physical inspection 
over 5 years. 

However, in July 2006, DHS announced that it had awarded contracts to 
develop and purchase $1.2 billion worth of ASP portal monitors over 5 
years. 

The ASP contract award has exceeded DNDO's estimate for total cost by 
about $200 million. This is before accounting for major aspects of the 
total cost, such as the costs of physical inspections of cargo, an 
additional procurement of 442 new PVT monitors, installation and 
integration costs, and maintenance costs. 

GAO's Conclusions and Recommendations: 

DNDO's analysis is incomplete and unreliable. 

* DNDO did not use its own test results and instead assumed relied on 
potential performance of the new equipment. 

* DNDO used inaccurate or incomplete cost data. 

* The analysis does not include all costs and benefits for acquiring 
and deploying portal monitors: 

Thus, DNDO's cost benefit analysis does not justify its recent decision 
to purchase and deploy ASP technology. 

DNDO should conduct a new cost-benefit analysis to justify the purchase 
of new radiation detection portal monitors by using sound analytical 
methods. 

DNDO should not spend any additional funds to purchase ASPs for 
deployment until it conducts realistic testing for both ASPs and PVTs 
that demonstrates that the new equipment provides superior detection 
and identification capabilities when compared to existing technology. 

Enclosure II: Comments from the Department of Homeland Security: 

Domestic Nuclear Detection Office: 
U.S. Department of Homeland Security: 
Washington, DC 20528: 

October 11, 2006: 

Memorandum For: Gene Aloise: 
Director, Natural Resources and Environment: 
Government Accountability Office: 

From: Vayl S. Oxford: 
Director: 
Domestic Nuclear Detection Office: 

Subject: Comments on GAO Report titled: Combating Nuclear Smuggling: 
DHS's Cost-Benefit Analysis to Support the Purchase of New Radiation 
Detection Portal Monitors Was Not Based on Available Performance Data 
and Did Not Fully Evaluate All the Monitors' Costs and Benefits, dated 
October 12, 2006: 

The Domestic Nuclear Detection Office (DNDO) has reviewed the subject 
report and the briefing presented to the House and Senate Committees on 
Appropriations dated August 21, 2006. Thank you very much for the 
opportunity to comment. While the DNDO has concerns with some of the 
report's findings, we remain committed to high fidelity testing prior 
to all full scale acquisition decisions, including the Advanced 
Spectroscopic Portal (ASP) program. 

The DNDO met on multiple occasions with the GAO staff to discuss the 
Cost Benefit Analysis (CBA) methodology, assumptions, data sources, and 
results. In all instances, we agreed with GAO's suggestions and worked 
to further refine the CBA, or provided written responses documenting 
the technical rationale for the DNDO's approach. However, some 
misunderstandings and/or disagreements remain between the DNDO and the 
GAO, including: 

* The GAO criticized the DNDO for assuming a probability of detection 
of 95%, even though initial test results did not show this same 
capability. However, the initial test results cited by the GAO were not 
intended to determine the absolute capabilities of deployed systems; 
rather, they were intended to support initial source selection 
decisions. Again, we remain committed to high fidelity testing prior to 
all full scale acquisition decisions. We plan to subject systems to 
additional testing prior to full scale production to evaluate their 
performance against stated specifications, including the requirement 
for 95% probability of detection. 

* The GAO stated that the DNDO tested the performance of PVT and ASP 
systems side-by-side, but did not use these results in the CBA. Again, 
the test series referenced was not intended to provide an objective 
side-by-side comparison. Further testing at both the Nevada Test Site 
and the New York Container Terminal will evaluate the validity of these 
assumptions. 

* The GAO stated that the CBA only evaluated systems' ability to detect 
highly enriched uranium (HEU) and did not consider other threats. DNDO 
agrees that threats other than HEU are equally important. However, we 
believe that HEU poses the greatest challenge from a detection 
standpoint, and serves as a reasonable threat baseline for the purpose 
of this analysis. 

The DNDO stands behind the basic conclusions of the CBA--that the 
Advanced Spectroscopic Portal (ASP) program is a sound investment for 
the USG and that the Joint DNDO-CBP Deployment Strategy, employing a 
mix of PVT and ASP systems for primary inspection and ASP systems for 
all secondary inspections is the optimal approach. 

DNDO believes that this report requires further collaboration and we 
look forward to continuing to work together on this important national 
priority. Please let me know if you or your staff need additional 
information; (202) 254-5688. 

Attachments: 

(360691): 

FOOTNOTES 

[1] The Departments of Energy, Defense, and State are also implementing 
programs to combat nuclear smuggling in other countries by providing 
radiation detection equipment and training to foreign border security 
personnel. 

[2] National Security Presidential Directive 43/Homeland Security 
Presidential Directive 14, Domestic Nuclear Detection (April 15, 2005). 
Congress recently passed a bill that would provide permanent statutory 
authority for DNDO and its activities. See Security and Accountability 
for Every Port Act of 2006, H.R. 4954,  501-502 (2006) ("SAFE Port 
Act"). As of October 12, 2006, the President had not yet signed the 
bill. 

[3] DOE manages the largest laboratory system of its kind in the world. 
The mission of DOE's 22 laboratories has evolved. Originally created to 
design and build atomic weapons, these laboratories have since expanded 
to conduct research in many disciplines-from high-energy physics to 
advanced computing. 

[4] Homeland Security Appropriations Act for Fiscal Year 2007, Pub. L. 
No. 109-295, tit. IV (2006). At the same time, Congress has directed 
that all containers entering the 22 busiest U.S. ports be scanned for 
radiation by December 31, 2007, and specified that to the extent 
practicable, DHS is to deploy "next generation radiation detection 
technology." See SAFE Port Act,  121. 

[5] GAO, Combating Nuclear Smuggling: DHS Has Made Progress Deploying 
Radiation Detection Equipment at U.S. Ports of Entry, but Concerns 
Remain, GAO-06-389 (Washington, D.C.: Mar. 22, 2006). 

[6] DNDO also issued a revised analysis in June 2006. 

[7] Federal Acquisition Regulation, 48 C.F.R.  15.404-1(b)(2)(v). 

[8] These test results involved identifying both HEU and depleted 
uranium. 

[9] These goals are now subject to the new requirements established by 
Congress in section 121 of the SAFE Ports Act. 

[10] Section 121 of the SAFE Ports Act reinforces this goal by 
requiring that DHS utilize next generation radiation technology "to the 
extent practicable", with conference report language noting that "such 
technology can reduce nuisance alarms" and "speed effective alarm 
resolution." See H.R. Conf. Rep. No. 109-711 (2006). 

[11] DHS and DOE are collaborating in building a new Radiological and 
Nuclear Countermeasures Test and Evaluation Complex at the Nevada Test 
Site to support the development, testing, acquisition, and deployment 
of radiation detection equipment. The facility is expected to become 
fully operational in early 2007. Currently, an interim facility at NTS 
is being used to test radiation detection equipment. 

[12] NIST did not evaluate the PVTs or compare their performance to the 
performance of the ASPs. 

[13] The ability to detect masked HEU is based on DOE guidance on 
performing the evaluation. 

[14] The performance specifications contain a requirement for 
detecting, not identifying, californium-252 with a 95 percent 
probability. Californium-252 has similar radiological properties to 
HEU. In addition, the performance specifications contain a requirement 
for detecting, but not identifying, other radiological materials such 
as cobalt-57, cobalt-60, barium-133, cesium-137, and americium-241. 

[15] Pacific Northwest National Laboratory, A Sensitivity Comparison of 
NaI and PVT Portal Monitors at a Land-Border Port-of-Entry, p. iii, 
November 2004. For Official Use Only 

[16] DHS, Capital Planning and Investment Control, Cost-Benefit 
Analysis (CBA) Guidebook 2006, Version 2.0, February 2006. Traditional 
rules of performing cost-benefit analyses include assessing the full 
life-cycle costs for operation and maintenance, and determining the 
level of confidence in cost data. 

[17] Enlisting Foreign Cooperation in U.S. Efforts to Prevent Nuclear 
Smuggling: Hearing Before the House Committee on Homeland Security, 
Subcommittee on Prevention of Nuclear and Biological Attack, 109TH 
Cong. (May 25, 2006) (statement of Mr. Vayl S. Oxford, Director, DNDO); 
Detecting Smuggled Nuclear Weapons, Hearing Before the Senate Judiciary 
Committee, Subcommittee on Terrorism, Technology, and Homeland 
Security, 109TH Cong. (July 27, 2006) (statement of Mr. Vayl S. Oxford, 
Director, DNDO). 

[18] DNDO, Cost Benefit Analysis for Next Generation Passive Radiation 
Detection of Cargo an the Nation's Border Crossings, May 30, 2006. 

[19] See GAO, Combating Nuclear Smuggling: DHS Has Made Progress 
Deploying Radiation Detection Equipment at U.S. Ports of Entry, but 
Concerns Remain, GAO-06-389 (Washington, D.C.: Mar 22, 2006). 

GAO's Mission: 

The Government Accountability Office, the investigative arm of 
Congress, exists to support Congress in meeting its constitutional 
responsibilities and to help improve the performance and accountability 
of the federal government for the American people. GAO examines the use 
of public funds; evaluates federal programs and policies; and provides 
analyses, recommendations, and other assistance to help Congress make 
informed oversight, policy, and funding decisions. GAO's commitment to 
good government is reflected in its core values of accountability, 
integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains 
abstracts and full-text files of current reports and testimony and an 
expanding archive of older products. The Web site features a search 
engine to help you locate documents using key words and phrases. You 
can print these documents in their entirety, including charts and other 
graphics. 

Each day, GAO issues a list of newly released reports, testimony, and 
correspondence. GAO posts this list, known as "Today's Reports," on its 
Web site daily. The list contains links to the full-text document 
files. To have GAO e-mail this list to you every afternoon, go to 
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order 
GAO Products" heading. 

Order by Mail or Phone: 

The first copy of each printed report is free. Additional copies are $2 
each. A check or money order should be made out to the Superintendent 
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 
more copies mailed to a single address are discounted 25 percent. 
Orders should be sent to: 

U.S. Government Accountability Office 

441 G Street NW, Room LM 

Washington, D.C. 20548: 

To order by Phone: 

Voice: (202) 512-6000: 

TDD: (202) 512-2537: 

Fax: (202) 512-6061: 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: www.gao.gov/fraudnet/fraudnet.htm 

E-mail: fraudnet@gao.gov 

Automated answering system: (800) 424-5454 or (202) 512-7470: 

Public Affairs: 

Jeff Nelligan, managing director, 

NelliganJ@gao.gov 

(202) 512-4800 

U.S. Government Accountability Office, 

441 G Street NW, Room 7149 

Washington, D.C. 20548: