FACE Recognition Technology:

FBI Should Better Ensure Privacy and Accuracy [Reissued on August 3, 2016]

GAO-16-267: Published: May 16, 2016. Publicly Released: Jun 15, 2016.

Multimedia:

Additional Materials:

Contact:

Diana Maurer
(202) 512-9627
maurerd@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

What GAO Found

The Department of Justice's (DOJ) Federal Bureau of Investigation (FBI) operates the Next Generation Identification-Interstate Photo System (NGI-IPS)— a face recognition service that allows law enforcement agencies to search a database of over 30 million photos to support criminal investigations. NGI-IPS users include the FBI and selected state and local law enforcement agencies, which can submit search requests to help identify an unknown person using, for example, a photo from a surveillance camera. When a state or local agency submits such a photo, NGI-IPS uses an automated process to return a list of 2 to 50 possible candidate photos from the database, depending on the user's specification. As of December 2015, the FBI has agreements with 7 states to search NGI-IPS, and is working with more states to grant access. In addition to the NGI-IPS, the FBI has an internal unit called Facial Analysis, Comparison and Evaluation (FACE) Services that provides face recognition capabilities, among other things, to support active FBI investigations. FACE Services not only has access to NGI-IPS, but can search or request to search databases owned by the Departments of State and Defense and 16 states, which use their own face recognition systems. Biometric analysts manually review photos before returning at most the top 1 or 2 photos as investigative leads to FBI agents.

DOJ developed a privacy impact assessment (PIA) of NGI-IPS in 2008, as required under the E-Government Act whenever agencies develop technologies that collect personal information. However, the FBI did not update the NGI-IPS PIA in a timely manner when the system underwent significant changes or publish a PIA for FACE Services before that unit began supporting FBI agents. DOJ ultimately approved PIAs for NGI-IPS and FACE Services in September and May 2015, respectively. The timely publishing of PIAs would provide the public with greater assurance that the FBI is evaluating risks to privacy when implementing systems. Similarly, NGI-IPS has been in place since 2011, but DOJ did not publish a System of Records Notice (SORN) that addresses the FBI's use of face recognition capabilities, as required by law, until May 5, 2016, after completion of GAO's review. The timely publishing of a SORN would improve the public's understanding of how NGI uses and protects personal information.

Prior to deploying NGI-IPS, the FBI conducted limited testing to evaluate whether face recognition searches returned matches to persons in the database (the detection rate) within a candidate list of 50, but has not assessed how often errors occur. FBI officials stated that they do not know, and have not tested, the detection rate for candidate list sizes smaller than 50, which users sometimes request from the FBI. By conducting tests to verify that NGI-IPS is accurate for all allowable candidate list sizes, the FBI would have more reasonable assurance that NGI-IPS provides leads that help enhance, rather than hinder, criminal investigations. Additionally, the FBI has not taken steps to determine whether the face recognition systems used by external partners, such as states and federal agencies, are sufficiently accurate for use by FACE Services to support FBI investigations. By taking such steps, the FBI could better ensure the data received from external partners is sufficiently accurate and do not unnecessarily include photos of innocent people as investigative leads.

Technology advancements have increased the overall accuracy of automated face recognition over the past few decades. According to the FBI, this technology can help law enforcement agencies identify criminals in their investigations.

GAO was asked to review the FBI's use of face recognition technology. This report examines: 1) the FBI's face recognition capabilities; and the extents to which 2) the FBI's use of face recognition adhered to privacy laws and policies and 3) the FBI assessed the accuracy of these capabilities.

To address these questions, GAO reviewed federal privacy laws, FBI policies, operating manuals, and other documentation on its face recognition capability. GAO interviewed officials from the FBI and other federal and two state agencies that coordinate with the FBI on face recognition.

What GAO Recommends

GAO is making six recommendations, including, that the Attorney General determine why PIAs and a SORN were not published as required and implement corrective actions, and for the FBI director to conduct tests to verify that NGI-IPS is accurate and take steps to determine whether systems used by external partners are sufficiently accurate for FBI's use. DOJ agreed with one, partially agreed with two, and disagreed with three of the six recommendations. In response, GAO clarified one recommendation, updated another recommendation, and continues to believe that all six recommendations remain valid as discussed further in this report.

For more information, contact Diana Maurer at (202) 512-9627 or maurerd@gao.gov.

Recommendations for Executive Action

  1. Status: Open

    Priority recommendation

    Comments: DOJ officials did not concur with this recommendation, and stated that the FBI has established practices that protect privacy and civil liberties beyond the requirements of the law. DOJ officials stated that it will internally evaluate the PIA process as part of the Department's overall commitment to improving its processes, not in response to our recommendation. In March 2017, we followed up with DOJ to obtain its current position on our recommendation. DOJ continues to believe that its approach in designing the NGI system was sufficient to meet legal privacy requirements and that our recommendation represents a "checkbox approach" to privacy. We disagree with DOJ's characterization of our recommendation. We continue to believe that the timely development and publishing of future PIAs would increase transparency of the department's systems. We recognize the steps the agency took to consider privacy protection during the development of the NGI system. We also stand by our position that notifying the public of these actions is important and provides the public with greater assurance that DOJ components are evaluating risks to privacy when implementing systems. As a result, the recommendation remains open and unimplemented.

    Recommendation: To improve transparency and better ensure that face recognition capabilities are being used in accordance with privacy protection laws and policy requirements, the Attorney General should assess the PIA development process to determine why PIAs were not published prior to using or updating face recognition capabilities, and implement corrective actions to ensure the timely development, updating, and publishing of PIAs before using or making changes to a system.

    Agency Affected: Department of Justice

  2. Status: Open

    Priority recommendation

    Comments: DOJ agreed, in part, with our recommendation and submitted the SORN for publication to the Federal Register on April 21, 2016, and it was published on May 5, 2016. DOJ did not agree that the publication of a SORN is required by law. We disagree with DOJ's interpretation regarding the legal requirements of a SORN. The Privacy Act of 1974 requires that when agencies establish or make changes to a system of records, they must notify the public through a SORN published in the Federal Register. DOJ's comments on our draft report acknowledge that the automated nature of face recognition technology and the sheer number of photos now available for searching raise important privacy and civil liberties considerations. DOJ officials also stated that the FBI's face recognition capabilities do not represent new collection, use, or sharing of personal information. We disagree. We believe that the ability to perform automated searches of millions of photos is fundamentally different in nature and scope than manual review of individual photos, and the potential impact on privacy is equally fundamentally different. By assessing the SORN development process and taking corrective actions to ensure timely development of future SORNs, the public would have a better understanding of how personal information is being used and protected by DOJ components. As a result, the recommendation remains open and unimplemented.

    Recommendation: To improve transparency and better ensure that face recognition capabilities are being used in accordance with privacy protection laws and policy requirements, the Attorney General should assess the SORN development process to determine why a SORN was not published that addressed the collection and maintenance of photos accessed and used through NGI for the FBI's face recognition capabilities prior to using NGI-IPS, and implement corrective actions to ensure SORNs are published before systems become operational.

    Agency Affected: Department of Justice

  3. Status: Open

    Priority recommendation

    Comments: In March 2017, DOJ provided us with the audit plan the CJIS Audit Unit developed in June 2016 for NGI-IPS users. In addition, DOJ reported that the CJIS Audit Unit began assessing NGI-IPS requirements at participating states in conjunction with its triennial National Identity Services audit and that, as of February 2017, the unit had conducted NGI-IPS audits of four states. Further, DOJ officials said CJIS developed an audit plan of the FACE Services Unit to coincide with the existing triennial FBI internal audit for 2018. However, DOJ did not provide the audit plan for the FACE Services Unit. DOJ officials said the methodology would be the same as the audit plan for NGI-IPS, but that methodology does not describe oversight on use of information obtained from external systems accessed by FACE Services employees. Therefore, we believe DOJ is making progress towards meeting the recommendation, but has not fully implemented our recommendation.

    Recommendation: To better ensure that face recognition capabilities are being used in accordance with privacy protection laws and policy requirements, the Director of the Federal Bureau of Investigation should conduct audits to determine the extent to which users of NGI-IPS and biometric images specialists in FACE Services are conducting face image searches in accordance with Criminal Justice Information Services Division policy requirements.

    Agency Affected: Department of Justice: Federal Bureau of Investigation

  4. Status: Open

    Priority recommendation

    Comments: In comments on our draft report in 2016, and reiterated during recommendation follow-up, as of March 2017, DOJ did not concur with this recommendation. DOJ officials stated that the FBI has performed accuracy testing to validate that the system meets the requirements for the detection rate, which fully satisfies requirements for the investigative lead service provided by NGI-IPS. We disagree with DOJ. A key focus of our recommendation is the need to ensure that NGI-IPS is sufficiently accurate for all allowable candidate list sizes. Although the FBI has tested the detection rate for a candidate list of 50 photos, NGI-IPS users are able to request smaller candidate lists (between 2 and 50 photos). FBI officials stated that they do not know, and have not tested, the detection rate for other candidate list sizes. According to these officials, a smaller candidate list would likely lower the detection rate because a smaller candidate list may not contain a likely match that would be present in a larger candidate list. However, according to the FBI Information Technology Life Cycle Management Directive, testing needs to confirm the system meets all user requirements. Because the accuracy of NGI-IPS's face recognition searches when returning fewer than 50 photos in a candidate list is unknown, the FBI is limited in understanding whether the results are accurate enough to meet NGI-IPS users' needs. DOJ officials also stated that searches of NGI-IPS produce a gallery of likely candidates to be used as investigative leads, not for positive identification. As a result, according to DOJ officials, NGI-IPS cannot produce false positives and there is no false positive rate for the system. We disagree with DOJ. The detection rate and the false positive rate are both necessary to assess the accuracy of a face recognition system. Generally, face recognition systems can be configured to allow for a greater or lesser number of matches. A greater number of matches would generally increase the detection rate, but would also increase the false positive rate. Similarly, a lesser number of matches would decrease the false positive rate, but would also decrease the detection rate. Reporting a detection rate of 86 percent without reporting the accompanying false positive rate presents an incomplete view of the system's accuracy. As a result, the recommendation remains open and unimplemented.

    Recommendation: To better ensure that face recognition systems are sufficiently accurate, the Director of the Federal Bureau of Investigation should conduct tests of NGI-IPS to verify that the system is sufficiently accurate for all allowable candidate list sizes, and ensure that the detection and false positive rate used in the tests are identified.

    Agency Affected: Department of Justice: Federal Bureau of Investigation

  5. Status: Open

    Priority recommendation

    Comments: As of March 2017, FBI officials stated they implemented the recommendation by submitting a paper to solicit feedback from users through the Fall 2016 Advisory Policy Board Process. Specifically, officials said the paper requested feedback on whether the face recognition searches of the NGI-IPS are meeting their needs, and input regarding search accuracy. According to FBI officials, no users expressed concern with any aspect of the NGI-IPS meeting their needs, including accuracy. Although FBI's action of providing working groups with a paper presenting GAO's recommendation is a step, the FBI's actions do not fully meet the recommendation. The FBI's paper was presented as informational, and did not result in any formal responses from users. We disagree with the FBI's conclusion that receiving no responses on the informational paper fulfills the operational review recommendation, which includes determining that NGI-IPS is meeting user's needs. As such, we continue to recommend the FBI conduct an operational review of NGI-IPS at least annually.

    Recommendation: To better ensure that face recognition systems are sufficiently accurate, the Director of the Federal Bureau of Investigation should conduct an operational review of NGI-IPS at least annually that includes an assessment of the accuracy of face recognition searches to determine if it is meeting federal, state, and local law enforcement needs and take actions, as necessary, to improve the system.

    Agency Affected: Department of Justice: Federal Bureau of Investigation

  6. Status: Open

    Priority recommendation

    Comments: In comments on our draft report in 2016, and reiterated during recommendation follow-up in 2017, DOJ officials did not concur with this recommendation and had no plans to implement it. DOJ officials stated that the FBI has no authority to set or enforce accuracy standards of face recognition technology operated by external agencies. In addition, DOJ officials stated that the FBI has implemented multiple layers of manual review that mitigate risks associated with the use of automated face recognition technology. Further, DOJ officials stated there is value in searching all available external databases, regardless of their level of accuracy. We disagree with the DOJ position. We continue to believe that the FBI should assess the quality of the data it is using from state and federal partners. We acknowledge that the FBI cannot and should not set accuracy standards for the face recognition systems used by external partners. We also do not dispute that the use of external face recognition systems by the FACE Services Unit could add value to FBI investigations. However, we disagree with FBI's assertion that no assessment of the quality of the data from state and federal partners is necessary. We also disagree with the DOJ assertion that manual review of automated search results is sufficient. Even with a manual review process, the FBI could miss investigative leads if a partner does not have a sufficiently accurate system. By relying on its external partners' face recognition systems, the FBI is using these systems as a component of its routine operations and is therefore responsible for ensuring the systems will help meet FBI's mission, goals and objectives. The recommendation remains open and unimplemented.

    Recommendation: To better ensure that face recognition systems are sufficiently accurate, the Director of the Federal Bureau of Investigation should take steps to determine whether each external face recognition system used by FACE Services is sufficiently accurate for the FBI's use and whether results from those systems should be used to support FBI investigations.

    Agency Affected: Department of Justice: Federal Bureau of Investigation

 

Explore the full database of GAO's Open Recommendations »

Jul 24, 2017

Jun 28, 2017

Apr 26, 2017

Apr 6, 2017

Mar 22, 2017

Mar 21, 2017

Feb 7, 2017

Dec 14, 2016

Nov 30, 2016

Looking for more? Browse all our products here