Data Mining:

DHS Needs to Improve Executive Oversight of Systems Supporting Counterterrorism

GAO-11-742, Sep 7, 2011

Additional Materials:

Contact:

David A. Powner
(202) 512-3000
contact@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

Data mining--a technique for extracting useful information from large volumes of data--is one type of analysis that the Department of Homeland Security (DHS) uses to help detect and prevent terrorist threats. While data-mining systems offer a number of promising benefits, their use also raises privacy concerns. GAO was asked to (1) assess DHS policies for evaluating the effectiveness and privacy protections of data-mining systems used for counterterrorism, (2) assess DHS agencies' efforts to evaluate the effectiveness and privacy protections of their data-mining systems, and (3) describe the challenges facing DHS in implementing an effective evaluation framework. To do so, GAO developed a systematic evaluation framework based on recommendations and best practices outlined by the National Research Council, industry practices, and prior GAO reports. GAO compared its evaluation framework to DHS's and three component agencies' policies and to six systems' practices, and interviewed agency officials about gaps in their evaluations and challenges..

As part of a systematic evaluation framework, agency policies should ensure organizational competence, evaluations of a system's effectiveness and privacy protections, executive review, and appropriate transparency throughout the system's life cycle. While DHS and three of its component agencies--U.S. Customs and Border Protection, U.S. Immigration and Customs Enforcement, and the U.S. Citizenship and Immigration Services--have established policies that address most of these key policy elements, the policies are not comprehensive. For example, DHS policies do not fully ensure executive review and transparency, and the component agencies' policies do not sufficiently require evaluating system effectiveness. DHS's Chief Information Officer reported that the agency is planning to improve its executive review process by conducting more intensive reviews of IT investments, including the data-mining systems reviewed in this report. Until such reforms are in place, DHS and its component agencies may not be able to ensure that critical data mining systems used in support of counterterrorism are both effective and that they protect personal privacy. Another aspect of a systematic evaluation framework involves ensuring that agencies implement sound practices for organizational competence, evaluations of a system's effectiveness and privacy protections, executive review, and appropriate transparency and oversight throughout a system's life cycle. Evaluations of six data mining systems from a mix of DHS component agencies showed that all six program offices took steps to evaluate their system's effectiveness and privacy protections. However, none performed all of the key activities associated with an effective evaluation framework. For example, four of the program offices executed most of the activities for evaluating program privacy impacts, but only one program office performed most of the activities related to obtaining executive review and approval. By not consistently performing necessary evaluations and reviews of these systems, DHS and its component agencies risk developing and acquiring systems that do not effectively support their agencies' missions and do not adequately ensure the protection of privacy-related information. DHS faces key challenges in implementing a framework to ensure systems are effective and provide privacy protections. These include reviewing and overseeing systems once they are in operation, stabilizing and implementing acquisition policies throughout the department, and ensuring that privacy-sensitive systems have timely and up-to-date privacy reviews. The shortfalls GAO noted in agency policies and practices provide insight into these challenges. Until DHS addresses these challenges, it will be limited in its ability to ensure that its systems have been adequately reviewed, are operating as intended, and are appropriately protecting individual privacy and assuring transparency to the public. GAO is recommending that DHS executives address gaps in agency evaluation policies and that component agency officials address shortfalls in their system evaluations. DHS concurred with GAO's recommendations and identified steps it is taking to address selected recommendations. The department also offered technical comments, which GAO incorporated as appropriate.

Status Legend:

More Info
  • Review Pending-GAO has not yet assessed implementation status.
  • Open-Actions to satisfy the intent of the recommendation have not been taken or are being planned, or actions that partially satisfy the intent of the recommendation have been taken.
  • Closed-implemented-Actions that satisfy the intent of the recommendation have been taken.
  • Closed-not implemented-While the intent of the recommendation has not been satisfied, time or circumstances have rendered the recommendation invalid.
    • Review Pending
    • Open
    • Closed - implemented
    • Closed - not implemented

    Recommendations for Executive Action

    Recommendation: In order to improve DHS's policies and practices for ensuring that datamining systems used for counterterrorism are effective and provide necessary privacy protections, the Secretary of Homeland Security should direct the Chief Privacy Officer to investigate whether the information sharing component of U.S. Immigration and Customs Enforcement (ICE) Pattern Analysis and Information Collection (ICEPIC), called the Law Enforcement Information Sharing Service, should be deactivated until a PIA that includes this component is approved.

    Agency Affected: Department of Homeland Security

    Status: Closed - Implemented

    Comments: The DHS Privacy Office began a privacy compliance review of the ICEPIC program in September 2011. While the review was ongoing, the Chief Privacy Officer reviewed, approved, and in October 2011, published an updated privacy impact assessment of the program that included the Law Enforcement Information Sharing service. In the results of the privacy compliance review, published in December 2011, the office noted that by directing the expedited preparation and review of the PIA update, the DHS Privacy Office was able to bring the system into compliance with the E-Government Act and DHS policy (which obviated the need to shut down the program). However, the report noted that the review highlighted the need for criteria to evaluate and remedy compliance gaps in a manner that upholds privacy and DHS operational needs. The office is now working to develop specific evaluation criteria and remedies to aid the Chief Privacy Officer when the office identifies compliance issues.

    Recommendation: In order to improve DHS's policies and practices for ensuring that datamining systems used for counterterrorism are effective and provide necessary privacy protections, the Secretary of Homeland Security should direct the Chief Privacy Officer to develop requirements for providing additional scrutiny of privacy protections for the sensitive information systems that are not transparent to the public through privacy impact assessments (PIAs).

    Agency Affected: Department of Homeland Security

    Status: Closed - Implemented

    Comments: In order to provide for additional scrutiny of the privacy protections for the sensitive information systems that do not have publicly-available privacy impact assessments, DHS noted in its agency comments letter that the DHS Privacy Office planned to include an annex on unreleased privacy impact assessments in its Annual Report to Congress that was marked and handled with the appropriate national security (or other sensitive, but unclassified) restrictions. The agency further noted that members of Congress could request the relevant documents or schedule a briefing with the agency after reviewing the annex. Subsequently, the Privacy Office included this annex in its most recent report to Congress, which will help ensure that systems that are not transparent to the public through privacy impact assessments get additional scrutiny. The Privacy Office further noted that it intends to include such annexes in all of its future Annual Reports to Congress. By completing this activity, the DHS Privacy Office can better assure it is maintaining appropriate transparency for its systems and of its actions.

    Recommendation: In order to improve DHS's policies and practices for ensuring that datamining systems used for counterterrorism are effective and provide necessary privacy protections, the Secretary of Homeland Security should direct the Chief Information Officer and Chief Procurement Officer to work with their counterparts at component agencies to identify steps to mitigate challenges related to the review and oversight of operational systems and to DHS's changing policy requirements and determine clear corrective actions, taking the impact on components and on individual program managers into account.

    Agency Affected: Department of Homeland Security

    Status: Open

    Comments: The Department of Homeland Security's Office of the Chief Information Officer (CIO) stated that it has taken several steps to improve the review and oversight of operational systems. For example, the office of the CIO reported that it revised its Operational Analysis guidance to incorporate additional guidance received from the Office of Management and Budget since the issuance of the previous guidance. The office further reported that the draft guidance has been sent to its component agencies for adoption and that they expect the guidance to be approved by the end of September 2012. We will continue to monitor the department's actions in this area.

    Recommendation: In order to improve DHS's policies and practices for ensuring that datamining systems used for counterterrorism are effective and provide necessary privacy protections, the Secretary of Homeland Security should direct the Chief Information Officer and Chief Procurement Officer to work with their counterparts at component agencies to ensure the consistency of component agencies' policies with DHS policies and proposed improvements to those policies, including requiring data quality assessments, requiring re-evaluations of operational systems, and establishing investment review boards with clearly defined structures for system review.

    Agency Affected: Department of Homeland Security

    Status: Open

    Comments: The Department of Homeland Security's Office of the Chief Information Officer (CIO) reported that it is starting to establish a process for reviewing component agencies' proposed IT directives and policies to ensure alignment with those issued by the department. The office further noted that is contacting the component agencies' CIOs to obtain and review existing policies in order to determine whether they are consistent with CIO corporate policy. If the IT policies are inconsistent, the department plans to work with the component counterparts to resolve the discrepancy. Further, the departmental CIO stated that the office plans to ensure that both department and component IT polices include requiring data quality assessments and re-evaluations of operational systems and establishing investment review boards. We will continue to monitor the agency's progress in implementing this recommendation.

    Recommendation: In order to improve DHS's policies and practices for ensuring that datamining systems used for counterterrorism are effective and provide necessary privacy protections, the Secretary of Homeland Security should direct the the appropriate component agency administrators to ensure that the system program offices for Analytical Framework for Intelligence (AFI), Automated Targeting System (ATS)-Passenger (ATS-P), Citizen and Immigration Data Repository (CIDR), Data Analysis and Research for Trade Transparency System (DARTTS), ICEPIC, and TECS Modernization (TECS-Mod) address the shortfalls in evaluating system effectiveness and privacy protections identified in this report, including shortfalls in applying acquisition practices, ensuring executive review and approval, and consistently documenting executive reviews.

    Agency Affected: Department of Homeland Security

    Status: Open

    Comments: We received updates on the systems we reviewed at three Department of Homeland Security (DHS) component agencies, including the Customs and Border Protection agency's Analytical Framework for Intelligence (AFI), Automated Targeting System-Passenger, and TECS-Mod system; the U.S. Citizenship and Immigration Service's Citizenship and Immigration Data Repository; and the Immigration and Customs Enforcement (ICE) agency's Data Analysis and Research for Trade Transparency System, and ICE Pattern Analysis and Information Collection (ICEPIC) system. AFI and ICEPIC have taken steps to complete privacy-related shortcomings, including by publishing new privacy impact assessments. However, none of the systems we reviewed have provided artifacts that demonstrate that have addressed all the shortfalls we identified in our report. We will continue to monitor DHS's activities in this area.