FACE Recognition Technology:

FBI Should Better Ensure Privacy and Accuracy [Reissued on August 3, 2016]

GAO-16-267: Published: May 16, 2016. Publicly Released: Jun 15, 2016.

Multimedia:

Additional Materials:

Contact:

Diana Maurer
(202) 512-9627
maurerd@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

What GAO Found

The Department of Justice's (DOJ) Federal Bureau of Investigation (FBI) operates the Next Generation Identification-Interstate Photo System (NGI-IPS)— a face recognition service that allows law enforcement agencies to search a database of over 30 million photos to support criminal investigations. NGI-IPS users include the FBI and selected state and local law enforcement agencies, which can submit search requests to help identify an unknown person using, for example, a photo from a surveillance camera. When a state or local agency submits such a photo, NGI-IPS uses an automated process to return a list of 2 to 50 possible candidate photos from the database, depending on the user's specification. As of December 2015, the FBI has agreements with 7 states to search NGI-IPS, and is working with more states to grant access. In addition to the NGI-IPS, the FBI has an internal unit called Facial Analysis, Comparison and Evaluation (FACE) Services that provides face recognition capabilities, among other things, to support active FBI investigations. FACE Services not only has access to NGI-IPS, but can search or request to search databases owned by the Departments of State and Defense and 16 states, which use their own face recognition systems. Biometric analysts manually review photos before returning at most the top 1 or 2 photos as investigative leads to FBI agents.

DOJ developed a privacy impact assessment (PIA) of NGI-IPS in 2008, as required under the E-Government Act whenever agencies develop technologies that collect personal information. However, the FBI did not update the NGI-IPS PIA in a timely manner when the system underwent significant changes or publish a PIA for FACE Services before that unit began supporting FBI agents. DOJ ultimately approved PIAs for NGI-IPS and FACE Services in September and May 2015, respectively. The timely publishing of PIAs would provide the public with greater assurance that the FBI is evaluating risks to privacy when implementing systems. Similarly, NGI-IPS has been in place since 2011, but DOJ did not publish a System of Records Notice (SORN) that addresses the FBI's use of face recognition capabilities, as required by law, until May 5, 2016, after completion of GAO's review. The timely publishing of a SORN would improve the public's understanding of how NGI uses and protects personal information.

Prior to deploying NGI-IPS, the FBI conducted limited testing to evaluate whether face recognition searches returned matches to persons in the database (the detection rate) within a candidate list of 50, but has not assessed how often errors occur. FBI officials stated that they do not know, and have not tested, the detection rate for candidate list sizes smaller than 50, which users sometimes request from the FBI. By conducting tests to verify that NGI-IPS is accurate for all allowable candidate list sizes, the FBI would have more reasonable assurance that NGI-IPS provides leads that help enhance, rather than hinder, criminal investigations. Additionally, the FBI has not taken steps to determine whether the face recognition systems used by external partners, such as states and federal agencies, are sufficiently accurate for use by FACE Services to support FBI investigations. By taking such steps, the FBI could better ensure the data received from external partners is sufficiently accurate and do not unnecessarily include photos of innocent people as investigative leads.

Technology advancements have increased the overall accuracy of automated face recognition over the past few decades. According to the FBI, this technology can help law enforcement agencies identify criminals in their investigations.

GAO was asked to review the FBI's use of face recognition technology. This report examines: 1) the FBI's face recognition capabilities; and the extents to which 2) the FBI's use of face recognition adhered to privacy laws and policies and 3) the FBI assessed the accuracy of these capabilities.

To address these questions, GAO reviewed federal privacy laws, FBI policies, operating manuals, and other documentation on its face recognition capability. GAO interviewed officials from the FBI and other federal and two state agencies that coordinate with the FBI on face recognition.

What GAO Recommends

GAO is making six recommendations, including, that the Attorney General determine why PIAs and a SORN were not published as required and implement corrective actions, and for the FBI director to conduct tests to verify that NGI-IPS is accurate and take steps to determine whether systems used by external partners are sufficiently accurate for FBI's use. DOJ agreed with one, partially agreed with two, and disagreed with three of the six recommendations. In response, GAO clarified one recommendation, updated another recommendation, and continues to believe that all six recommendations remain valid as discussed further in this report.

For more information, contact Diana Maurer at (202) 512-9627 or maurerd@gao.gov.

Recommendations for Executive Action

  1. Status: Open

    Priority recommendation

    Comments: DOJ officials did not concur with this recommendation, and stated that the FBI has established practices that protect privacy and civil liberties beyond the requirements of the law. DOJ officials stated that it will internally evaluate the PIA process as part of the department's overall commitment to improving its processes, not in response to our recommendation. In November 2018, DOJ officials told us that they have reviewed the PIA development process and determined that one reason that the FBI's face recognition PIAs were not completed more quickly was because the FBI and DOJ engaged in an extensive PIA revision process. As a result, DOJ officials stated that they implemented a pilot to expedite the PIA approval process, which included developing a PIA approval template and focusing the review solely on legal sufficiency instead of a more comprehensive review that included less significant editorial changes. While we believe this is a good first step, DOJ reported that, in some circumstances, DOJ will approve the PIA assessment and allow the FBI to move forward with full operations of the information system while finalizing the PIA document for publication. In this scenario, the public would remain unaware of the department's considerations of privacy. To fully implement this recommendation, DOJ should ensure that its procedures require PIAs to be published prior to the operation of a system where practicable (i.e., the PIA does not describe a classified system), complete the pilot, assess the results, and, if determined to be a success, institutionalize the changes.

    Recommendation: To improve transparency and better ensure that face recognition capabilities are being used in accordance with privacy protection laws and policy requirements, the Attorney General should assess the PIA development process to determine why PIAs were not published prior to using or updating face recognition capabilities, and implement corrective actions to ensure the timely development, updating, and publishing of PIAs before using or making changes to a system.

    Agency Affected: Department of Justice

  2. Status: Open

    Priority recommendation

    Comments: DOJ agreed, in part, with our recommendation and submitted the SORN for publication to the Federal Register on April 21, 2016, and it was published on May 5, 2016. However, DOJ disagrees that it was required as a matter of law to file a revised SORN. According to DOJ, it continues to review and update its pre-existing SORNs on an ongoing basis and is continually improving the scope and efficiency of its privacy processes. As of November 2018, DOJ has not taken actions to address our recommendation. However, if the PIA pilot is deemed successful, DOJ reported that the FBI will extend the concept of the pilot to the preparation of SORNs. We will continue to monitor these efforts to determine the extent to which DOJ's pilot helps ensure SORNs are published before systems become operational.

    Recommendation: To improve transparency and better ensure that face recognition capabilities are being used in accordance with privacy protection laws and policy requirements, the Attorney General should assess the SORN development process to determine why a SORN was not published that addressed the collection and maintenance of photos accessed and used through NGI for the FBI's face recognition capabilities prior to using NGI-IPS, and implement corrective actions to ensure SORNs are published before systems become operational.

    Agency Affected: Department of Justice

  3. Status: Closed - Implemented

    Priority recommendation

    Comments: In March 2017, DOJ provided us with the audit plan the CJIS Audit Unit developed in June 2016 for NGI-IPS users. In February 2018, DOJ officials stated that they have conducted eight NGI-IPS audits, which have found no significant findings of noncompliance. DOJ also provided us with copies of the final audit results for one state and its audit NGI-IPS reference guide. Further, DOJ officials said CJIS developed an audit plan of the FACE Services and completed an initial audit in September 2018.The FBI reported that it finalized the audit report in April 2019, which concluded that Face Services is operating in accordance with privacy laws and policies. Further, the FBI reported in May 2019 that audits of FACE Services will continue on a tri-annual basis and that it conducts tri-annual audits of states that use NGI-IPS. As a result, DOJ has fully implemented our recommendation.

    Recommendation: To better ensure that face recognition capabilities are being used in accordance with privacy protection laws and policy requirements, the Director of the Federal Bureau of Investigation should conduct audits to determine the extent to which users of NGI-IPS and biometric images specialists in FACE Services are conducting face image searches in accordance with Criminal Justice Information Services Division policy requirements.

    Agency Affected: Department of Justice: Federal Bureau of Investigation

  4. Status: Open

    Priority recommendation

    Comments: In comments on our draft report in 2016, and reiterated during recommendation follow-up, DOJ did not concur with this recommendation. As of November 2018, DOJ has not taken action to address our recommendation. DOJ officials stated that the FBI has performed accuracy testing to validate that the system meets the requirements for the detection rate, which fully satisfies requirements for the investigative lead service provided by NGI-IPS. We disagree with DOJ. A key focus of our recommendation is the need to ensure that NGI-IPS is sufficiently accurate for all allowable candidate list sizes. Although the FBI has tested the detection rate for a candidate list of 50 photos, NGI-IPS users are able to request smaller candidate lists (between 2 and 50 photos). FBI officials stated that they do not know, and have not tested, the detection rate for other candidate list sizes. According to these officials, a smaller candidate list would likely lower the detection rate because a smaller candidate list may not contain a likely match that would be present in a larger candidate list. However, according to the FBI Information Technology Life Cycle Management Directive, testing needs to confirm the system meets all user requirements. Because the accuracy of NGI-IPS's face recognition searches when returning fewer than 50 photos in a candidate list is unknown, the FBI is limited in understanding whether the results are accurate enough to meet NGI-IPS users' needs. DOJ officials also stated that searches of NGI-IPS produce a gallery of likely candidates to be used as investigative leads, not for positive identification. As a result, according to DOJ officials, NGI-IPS cannot produce false positives and there is no false positive rate for the system. We disagree with DOJ. The detection rate and the false positive rate are both necessary to assess the accuracy of a face recognition system. Generally, face recognition systems can be configured to allow for a greater or lesser number of matches. A greater number of matches would generally increase the detection rate, but would also increase the false positive rate. Similarly, a lesser number of matches would decrease the false positive rate, but would also decrease the detection rate. Reporting a detection rate of 86 percent without reporting the accompanying false positive rate presents an incomplete view of the system's accuracy. As a result, the recommendation remains open and unimplemented.

    Recommendation: To better ensure that face recognition systems are sufficiently accurate, the Director of the Federal Bureau of Investigation should conduct tests of NGI-IPS to verify that the system is sufficiently accurate for all allowable candidate list sizes, and ensure that the detection and false positive rate used in the tests are identified.

    Agency Affected: Department of Justice: Federal Bureau of Investigation

  5. Status: Open

    Priority recommendation

    Comments: As of March 2017, FBI officials stated they implemented the recommendation by submitting a paper to solicit feedback from users through the Fall 2016 Advisory Policy Board Process. Specifically, officials said the paper requested feedback on whether the face recognition searches of the NGI-IPS are meeting their needs, and input regarding search accuracy. FBI officials reported in February 2018 that they repeated this process in the fall of 2017. According to FBI officials, no users expressed concern with any aspect of the NGI-IPS meeting their needs, including accuracy. Although FBI's action of providing working groups with a paper presenting GAO's recommendation is a step, the FBI's actions do not fully meet the recommendation. The FBI's paper was presented as informational, and did not result in any formal responses from users. We disagree with the FBI's conclusion that receiving no responses on the informational paper fulfills the operational review recommendation, which includes determining that NGI-IPS is meeting user's needs. As of November 2018, DOJ has not taken any additional action to address this recommendation. As such, we continue to recommend the FBI conduct an operational review of NGI-IPS at least annually.

    Recommendation: To better ensure that face recognition systems are sufficiently accurate, the Director of the Federal Bureau of Investigation should conduct an operational review of NGI-IPS at least annually that includes an assessment of the accuracy of face recognition searches to determine if it is meeting federal, state, and local law enforcement needs and take actions, as necessary, to improve the system.

    Agency Affected: Department of Justice: Federal Bureau of Investigation

  6. Status: Open

    Priority recommendation

    Comments: In comments on our draft report in 2016, and reiterated during recommendation follow-up in February and November 2018, DOJ officials did not concur with this recommendation and had no plans to implement it. DOJ officials stated that the FBI has no authority to set or enforce accuracy standards of face recognition technology operated by external agencies. In addition, DOJ officials stated that the FBI has implemented multiple layers of manual review that mitigate risks associated with the use of automated face recognition technology. Further, DOJ officials stated there is value in searching all available external databases, regardless of their level of accuracy. We disagree with the DOJ position. We continue to believe that the FBI should assess the quality of the data it is using from state and federal partners. We acknowledge that the FBI cannot and should not set accuracy standards for the face recognition systems used by external partners. We also do not dispute that the use of external face recognition systems by the FACE Services Unit could add value to FBI investigations. However, we disagree with FBI's assertion that no assessment of the quality of the data from state and federal partners is necessary. We also disagree with the DOJ assertion that manual review of automated search results is sufficient. Even with a manual review process, the FBI could miss investigative leads if a partner does not have a sufficiently accurate system. By relying on its external partners' face recognition systems, the FBI is using these systems as a component of its routine operations and is therefore responsible for ensuring the systems will help meet FBI's mission, goals and objectives. The recommendation remains open and unimplemented.

    Recommendation: To better ensure that face recognition systems are sufficiently accurate, the Director of the Federal Bureau of Investigation should take steps to determine whether each external face recognition system used by FACE Services is sufficiently accurate for the FBI's use and whether results from those systems should be used to support FBI investigations.

    Agency Affected: Department of Justice: Federal Bureau of Investigation

 

Explore the full database of GAO's Open Recommendations »

Jul 9, 2019

Jun 27, 2019

Jun 7, 2019

Jun 4, 2019

May 30, 2019

May 9, 2019

Apr 17, 2019

  • justice icon, source: Comstock

    Priority Open Recommendations:

    Department of Justice
    GAO-19-361SP: Published: Apr 10, 2019. Publicly Released: Apr 17, 2019.

Mar 21, 2019

Mar 7, 2019

Feb 12, 2019

Looking for more? Browse all our products here