FACE Recognition Technology: FBI Should Better Ensure Privacy and Accuracy [Reissued on August 3, 2016]
Highlights
What GAO Found
The Department of Justice's (DOJ) Federal Bureau of Investigation (FBI) operates the Next Generation Identification-Interstate Photo System (NGI-IPS)— a face recognition service that allows law enforcement agencies to search a database of over 30 million photos to support criminal investigations. NGI-IPS users include the FBI and selected state and local law enforcement agencies, which can submit search requests to help identify an unknown person using, for example, a photo from a surveillance camera. When a state or local agency submits such a photo, NGI-IPS uses an automated process to return a list of 2 to 50 possible candidate photos from the database, depending on the user's specification. As of December 2015, the FBI has agreements with 7 states to search NGI-IPS, and is working with more states to grant access. In addition to the NGI-IPS, the FBI has an internal unit called Facial Analysis, Comparison and Evaluation (FACE) Services that provides face recognition capabilities, among other things, to support active FBI investigations. FACE Services not only has access to NGI-IPS, but can search or request to search databases owned by the Departments of State and Defense and 16 states, which use their own face recognition systems. Biometric analysts manually review photos before returning at most the top 1 or 2 photos as investigative leads to FBI agents.
DOJ developed a privacy impact assessment (PIA) of NGI-IPS in 2008, as required under the E-Government Act whenever agencies develop technologies that collect personal information. However, the FBI did not update the NGI-IPS PIA in a timely manner when the system underwent significant changes or publish a PIA for FACE Services before that unit began supporting FBI agents. DOJ ultimately approved PIAs for NGI-IPS and FACE Services in September and May 2015, respectively. The timely publishing of PIAs would provide the public with greater assurance that the FBI is evaluating risks to privacy when implementing systems. Similarly, NGI-IPS has been in place since 2011, but DOJ did not publish a System of Records Notice (SORN) that addresses the FBI's use of face recognition capabilities, as required by law, until May 5, 2016, after completion of GAO's review. The timely publishing of a SORN would improve the public's understanding of how NGI uses and protects personal information.
Prior to deploying NGI-IPS, the FBI conducted limited testing to evaluate whether face recognition searches returned matches to persons in the database (the detection rate) within a candidate list of 50, but has not assessed how often errors occur. FBI officials stated that they do not know, and have not tested, the detection rate for candidate list sizes smaller than 50, which users sometimes request from the FBI. By conducting tests to verify that NGI-IPS is accurate for all allowable candidate list sizes, the FBI would have more reasonable assurance that NGI-IPS provides leads that help enhance, rather than hinder, criminal investigations. Additionally, the FBI has not taken steps to determine whether the face recognition systems used by external partners, such as states and federal agencies, are sufficiently accurate for use by FACE Services to support FBI investigations. By taking such steps, the FBI could better ensure the data received from external partners is sufficiently accurate and do not unnecessarily include photos of innocent people as investigative leads.
Why GAO Did This Study
Technology advancements have increased the overall accuracy of automated face recognition over the past few decades. According to the FBI, this technology can help law enforcement agencies identify criminals in their investigations.
GAO was asked to review the FBI's use of face recognition technology. This report examines: 1) the FBI's face recognition capabilities; and the extents to which 2) the FBI's use of face recognition adhered to privacy laws and policies and 3) the FBI assessed the accuracy of these capabilities.
To address these questions, GAO reviewed federal privacy laws, FBI policies, operating manuals, and other documentation on its face recognition capability. GAO interviewed officials from the FBI and other federal and two state agencies that coordinate with the FBI on face recognition.
Reissued on August 3, 2016
Recommendations
GAO is making six recommendations, including, that the Attorney General determine why PIAs and a SORN were not published as required and implement corrective actions, and for the FBI director to conduct tests to verify that NGI-IPS is accurate and take steps to determine whether systems used by external partners are sufficiently accurate for FBI's use. DOJ agreed with one, partially agreed with two, and disagreed with three of the six recommendations. In response, GAO clarified one recommendation, updated another recommendation, and continues to believe that all six recommendations remain valid as discussed further in this report.
Recommendations for Executive Action
| Agency Affected | Recommendation | Status |
|---|---|---|
| Department of Justice |
Priority Rec.
To improve transparency and better ensure that face recognition capabilities are being used in accordance with privacy protection laws and policy requirements, the Attorney General should assess the PIA development process to determine why PIAs were not published prior to using or updating face recognition capabilities, and implement corrective actions to ensure the timely development, updating, and publishing of PIAs before using or making changes to a system. |
In November 2018, DOJ officials told us that they had reviewed the PIA development process and determined that one reason that the FBI's face recognition PIAs were not completed more quickly was because the FBI and DOJ engaged in an extensive PIA revision process. In response, DOJ officials stated that they implemented a pilot to expedite the FBI PIA approval process, which included developing a PIA approval template and focusing the review on legal sufficiency instead of a more comprehensive review that included less significant editorial changes. According to DOJ and FBI officials, the pilot was a success and resulted in the ability to approve a greater number of PIAs in less time. DOJ reported that between April 2018, when DOJ began to implement the pilot, and April 2019, DOJ approved nineteen FBI PIAs, in contrast to the previous 12-month period, when DOJ approved only five FBI PIAs. From May 2019 to July 2019, DOJ approved 12 additional PIAs. In July 2019, DOJ formalized the expedited review process for FBI PIAs in a memorandum for the record issued by the DOJ Office of the Deputy Attorney General, Office of Privacy and Civil Liberties (OPCL). The memorandum states that DOJ is to provide comments or approve a PIA within 30 days of the FBI submitting a draft PIA to DOJ OPCL for review. Further, the memorandum provides that DOJ will collaborate with FBI to maximize transparency and make PIAs publicly available, if practicable, prior to operation of a system. DOJ officials also stated in June 2019 that based on the success of the FBI PIA pilot, the department will look to explore the feasibility of utilizing a similar expedited review process in the development and publication of PIAs issued by other DOJ components. The timely publishing of PIAs will provide the public with greater assurance that the FBI is evaluating risks to privacy when implementing systems.
|
| Department of Justice |
Priority Rec.
To improve transparency and better ensure that face recognition capabilities are being used in accordance with privacy protection laws and policy requirements, the Attorney General should assess the SORN development process to determine why a SORN was not published that addressed the collection and maintenance of photos accessed and used through NGI for the FBI's face recognition capabilities prior to using NGI-IPS, and implement corrective actions to ensure SORNs are published before systems become operational. |
In April 2018, in response to a separate GAO recommendation, DOJ implemented a pilot to expedite the FBI Privacy Impact Assessment approval process, and, in June 2019, it applied this process to how the department develops and reviews SORNs, according to DOJ officials. These officials stated that the pilot focused the review on legal sufficiency instead of a more comprehensive review that included less significant editorial changes. Further, DOJ documented in its July 2019 memorandum for the record the requirement that the DOJ Office of Privacy and Civil Liberties (OPCL) is to provide comments or approval within 30 days of the FBI submitting a draft SORN to DOJ OPCL for review. In addition, the memorandum provides that after DOJ approves an FBI SORN, it will begin the formal publication process, in accordance with the law and Office of Management and Budget (OMB) policy. DOJ and FBI officials also stated that it is their goal to consistently publish SORNs before a system's implementation. DOJ officials told us that the intent of the July 2019 memorandum was to make changes to the current process that would permit DOJ to more efficiently and effectively facilitate all aspects of SORN development within its purview prior to submission to OMB. In July 2020, DHS provided documentation that the changes documented in the memorandum resulted in DOJ increasing the number of FBI-related SORNs it has published. Based on DOJ records, from June 2018 to June 2019, DOJ published only one FBI-related SORN in the Federal Register. From June 2019 to January 2020, DOJ records indicate that it had published four FBI-related SORNS in the Federal Register. By taking such steps, DOJ has met the intent of the recommendation. The timely publishing of SORNs helps improve the public's understanding of how NGI uses and protects personal information.
|
| Federal Bureau of Investigation |
Priority Rec.
To better ensure that face recognition capabilities are being used in accordance with privacy protection laws and policy requirements, the Director of the Federal Bureau of Investigation should conduct audits to determine the extent to which users of NGI-IPS and biometric images specialists in FACE Services are conducting face image searches in accordance with Criminal Justice Information Services Division policy requirements. |
In March 2017, DOJ provided us with the audit plan the CJIS Audit Unit developed in June 2016 for NGI-IPS users. In February 2018, DOJ officials stated that they have conducted eight NGI-IPS audits, which have found no significant findings of noncompliance. DOJ also provided us with copies of the final audit results for one state and its audit NGI-IPS reference guide. Further, DOJ officials said CJIS developed an audit plan of the FACE Services and completed an initial audit in September 2018.The FBI reported that it finalized the audit report in April 2019, which concluded that Face Services is operating in accordance with privacy laws and policies. Further, the FBI reported in May 2019 that audits of FACE Services will continue on a tri-annual basis and that it conducts tri-annual audits of states that use NGI-IPS. As a result, DOJ has fully implemented our recommendation.
|
| Federal Bureau of Investigation |
Priority Rec.
To better ensure that face recognition systems are sufficiently accurate, the Director of the Federal Bureau of Investigation should conduct tests of NGI-IPS to verify that the system is sufficiently accurate for all allowable candidate list sizes, and ensure that the detection and false positive rate used in the tests are identified. |
In response, in June 2019, the FBI provided information on candidate list size testing. In 2017, the FBI tasked a federally funded research center to assess the NGI-IPS system against operational parameters, including the length of candidate lists. In December 2017, the center reported that the NGI-IPS system successfully met the detection rate for all candidate sizes between 2 and 50. The report did not specifically assess how often NGI-IPS face recognition searches erroneously matched persons to the database, but it characterized the impact that database size would have on false positives. Further, the test conducted is consistent with a law enforcement scenario in which a human reviewer is employed to review the candidates returned from an identification search, according to the National Institute of Standards and Technology. FBI officials told us that FBI image specialists review all images that are returned. These officials also stated that they provide training to state and local users of NGI-IPS. By conducting tests to verify that NGI-IPS is sufficiently accurate for all allowable candidate list sizes and adjudicating all candidate photos returned by NGI-IPS, the FBI has better assurance that NGI-IPS provides investigative leads that help enhance, rather than hinder or overly burden, criminal investigation work.
|
| Federal Bureau of Investigation |
Priority Rec.
To better ensure that face recognition systems are sufficiently accurate, the Director of the Federal Bureau of Investigation should conduct an operational review of NGI-IPS at least annually that includes an assessment of the accuracy of face recognition searches to determine if it is meeting federal, state, and local law enforcement needs and take actions, as necessary, to improve the system. |
In July 2019, the FBI reported that it had designed a Face Recognition Test Strategy for Operational Analysis with the objective of being able to perform an annual operational review of NGI-IPS. To complete the testing, the FBI developed a biometric evaluation tool that operationally evaluates the face recognition capabilities of NGI-IPS. According to the FBI, this tool's purpose is to evaluate the operational integrity of NGI-IPS as change is introduced over time. Examples of changes may include architectural design (infrastructure or code), new or removed photos, and algorithm enhancements. The FBI reported that this testing is performed by taking a known data set, or a single operational day, and performing the same searches at different points in time (as the system and data changes) and evaluating the results for consistency. The FBI reported that it conducted these operational tests in January 2020 and April 2020. According to the FBI, the testing shows consistent NGI-IPS results between the two timeframes. The FBI reported that it plans to continue these searches on a quarterly basis. In addition, since fall 2016, the FBI has submitted annual staff papers through their Criminal Justice Information Services Division Advisory Policy Board process to solicit feedback from its users regarding search accuracy, according to FBI officials. The FBI reported that no users have expressed any concern with the operation or accuracy of NGI-IPS, as of July 2020. By taking such steps, the FBI has met the intent of the recommendation and now has more assurance that NGI-IPS is operating as intended and is meeting users' needs.
|
| Federal Bureau of Investigation |
Priority Rec.
To better ensure that face recognition systems are sufficiently accurate, the Director of the Federal Bureau of Investigation should take steps to determine whether each external face recognition system used by FACE Services is sufficiently accurate for the FBI's use and whether results from those systems should be used to support FBI investigations. |
In June 2019, FBI officials stated that they plan to survey state partners about the face recognition technology they use and submitted the survey to the Office of Management and Budget for approval. According to FBI officials, they hoped to gain insights into the technology the states use, which could provide them with information on the accuracy of the state systems. We reviewed the survey questions and agreed that the survey would provide the FBI with information that can help the bureau determine if those systems are sufficiently accurate for its use. In August 2019, FBI officials reported that 13 of the 21 state agencies that partner with the FBI FACE Services Unit for face recognition searches responded to the survey, with none of the responses raising concerns with the accuracy of the states' face recognition systems, according to FBI officials. FBI officials stated that they also requested that federal agencies that partner with FACE Services complete the survey. The Department of Defense responded to the survey, according to FBI documentation. By taking such steps, the FBI has more assurance that the data received from external partners is sufficiently accurate for FBI's use and do not unnecessarily include photos of innocent people as investigative leads. Therefore, this recommendation is closed as implemented.
|