What GAO Found

After the 2001 Anthrax attacks, the genetic tests that were conducted by the Federal Bureau of Investigation's (FBI) four contractors were generally scientifically verified and validated, and met the FBI's criteria. However, GAO found that the FBI lacked a comprehensive approach—or framework—that could have ensured standardization of the testing process. As a result, each of the contractors developed their tests differently, and one contractor did not conduct verification testing, a key step in determining whether a test will meet a user's requirements, such as for sensitivity or accuracy. Also, GAO found that the contractors did not develop the level of statistical confidence for interpreting the testing results for the validation tests they performed. Responses to future incidents could be improved by using a standardized framework for achieving minimum performance standards during verification and validation, and by incorporating statistical analyses when interpreting validation testing results.

GAO identified six characteristics of a statistical framework that can be applied for analyzing scientific evidence. When GAO compared the approach the FBI used to this framework, it found that that the FBI's approach could have been improved in three of six areas. First, the FBI's research did not provide a full understanding of the methods and conditions that give rise to genetic mutations used to differentiate between samples of B. anthracis. Second, the FBI did not institute rigorous controls over the sampling procedures it used to build the repository of B. anthracis samples. Third, the FBI did not include measures of uncertainty to strengthen the interpretation of the scientific evidence. GAO found that since 2001 the FBI has taken some steps to build formal forensic statistical expertise. The FBI's approach to future incidents could benefit from including such expertise early in an investigation.

The lack of an understanding of how bacteria change (mutate) in their natural environment and in a laboratory is a key scientific gap that remains and could affect testing conducted in future incidents. Specifically, the significance of using such mutations as genetic markers for analyzing evidentiary samples to determine their origins is not clear. This gap affects both the development of genetic tests targeting such mutations and statistical analyses of the results of their use on evidentiary samples. The Department of Homeland Security is currently funding some research on genetic changes in bacteria and genome sequencing methods, among others. Such research is a step in the right direction since the FBI is planning to use genome sequencing methods in future investigations. However, because this research may not be complete for several more years, the extent to which it will close this gap is not known.

Why GAO Did This Study

In 2001, the FBI investigated an intentional release of B. anthracis, a bacterium that causes anthrax, which was identified as the Ames strain. Subsequently, FBI contractors developed and validated several genetic tests to analyze B. anthracis samples for the presence of certain genetic mutations. The FBI had previously collected and maintained these samples in a repository.

GAO was asked to review the FBI's genetic test development process and statistical analyses. This report addresses (1) the extent to which these genetic tests were scientifically verified and validated; (2) the characteristics of an adequate statistical approach for analyzing samples, whether the approach used was adequate, and how it could be improved for future efforts; and (3) whether any remaining scientific concerns regarding the validation of genetic tests and statistical approaches need to be addressed for future analyses. GAO reviewed agency and contractor documentation, conducted literature reviews, and conducted statistical analyses of the repository data. GAO's review focused solely on two aspects of the FBI's scientific evidence: the validation of the genetic tests and the statistical approach for the analyses of the results. GAO did not review and is not taking a position on the conclusions the FBI reached when it closed its investigation in 2010.

What GAO Recommends

GAO recommends that the FBI develop a framework for validation and statistical approaches for future investigations. The FBI agreed with our recommendations.

For more information, contact Timothy M. Persons, Chief Scientist, at (202) 512-6412 or personst@gao.gov.

• View Report (PDF)