Data Mining:

DHS Needs to Improve Executive Oversight of Systems Supporting Counterterrorism

GAO-11-742: Published: Sep 7, 2011. Publicly Released: Oct 7, 2011.

Additional Materials:

Contact:

David A. Powner
(202) 512-3000
contact@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

Data mining--a technique for extracting useful information from large volumes of data--is one type of analysis that the Department of Homeland Security (DHS) uses to help detect and prevent terrorist threats. While data-mining systems offer a number of promising benefits, their use also raises privacy concerns. GAO was asked to (1) assess DHS policies for evaluating the effectiveness and privacy protections of data-mining systems used for counterterrorism, (2) assess DHS agencies' efforts to evaluate the effectiveness and privacy protections of their data-mining systems, and (3) describe the challenges facing DHS in implementing an effective evaluation framework. To do so, GAO developed a systematic evaluation framework based on recommendations and best practices outlined by the National Research Council, industry practices, and prior GAO reports. GAO compared its evaluation framework to DHS's and three component agencies' policies and to six systems' practices, and interviewed agency officials about gaps in their evaluations and challenges..

As part of a systematic evaluation framework, agency policies should ensure organizational competence, evaluations of a system's effectiveness and privacy protections, executive review, and appropriate transparency throughout the system's life cycle. While DHS and three of its component agencies--U.S. Customs and Border Protection, U.S. Immigration and Customs Enforcement, and the U.S. Citizenship and Immigration Services--have established policies that address most of these key policy elements, the policies are not comprehensive. For example, DHS policies do not fully ensure executive review and transparency, and the component agencies' policies do not sufficiently require evaluating system effectiveness. DHS's Chief Information Officer reported that the agency is planning to improve its executive review process by conducting more intensive reviews of IT investments, including the data-mining systems reviewed in this report. Until such reforms are in place, DHS and its component agencies may not be able to ensure that critical data mining systems used in support of counterterrorism are both effective and that they protect personal privacy. Another aspect of a systematic evaluation framework involves ensuring that agencies implement sound practices for organizational competence, evaluations of a system's effectiveness and privacy protections, executive review, and appropriate transparency and oversight throughout a system's life cycle. Evaluations of six data mining systems from a mix of DHS component agencies showed that all six program offices took steps to evaluate their system's effectiveness and privacy protections. However, none performed all of the key activities associated with an effective evaluation framework. For example, four of the program offices executed most of the activities for evaluating program privacy impacts, but only one program office performed most of the activities related to obtaining executive review and approval. By not consistently performing necessary evaluations and reviews of these systems, DHS and its component agencies risk developing and acquiring systems that do not effectively support their agencies' missions and do not adequately ensure the protection of privacy-related information. DHS faces key challenges in implementing a framework to ensure systems are effective and provide privacy protections. These include reviewing and overseeing systems once they are in operation, stabilizing and implementing acquisition policies throughout the department, and ensuring that privacy-sensitive systems have timely and up-to-date privacy reviews. The shortfalls GAO noted in agency policies and practices provide insight into these challenges. Until DHS addresses these challenges, it will be limited in its ability to ensure that its systems have been adequately reviewed, are operating as intended, and are appropriately protecting individual privacy and assuring transparency to the public. GAO is recommending that DHS executives address gaps in agency evaluation policies and that component agency officials address shortfalls in their system evaluations. DHS concurred with GAO's recommendations and identified steps it is taking to address selected recommendations. The department also offered technical comments, which GAO incorporated as appropriate.

Recommendations for Executive Action

  1. Status: Closed - Implemented

    Comments: The Department of Homeland Security's (DHS) Office of the Chief Information Officer (OCIO) established a policy in February 2014 that directs component chief information officers to ensure that DHS's components comply with Department-level IT policies and directives. It also identified agency officials at each of its component agencies and completed a review of each agency's key policies to ensure that the policies align with DHS policies.

    Recommendation: In order to improve DHS's policies and practices for ensuring that datamining systems used for counterterrorism are effective and provide necessary privacy protections, the Secretary of Homeland Security should direct the Chief Information Officer and Chief Procurement Officer to work with their counterparts at component agencies to ensure the consistency of component agencies' policies with DHS policies and proposed improvements to those policies, including requiring data quality assessments, requiring re-evaluations of operational systems, and establishing investment review boards with clearly defined structures for system review.

    Agency Affected: Department of Homeland Security

  2. Status: Closed - Implemented

    Comments: The Department of Homeland Security's (DHS) Office of the Chief Information Officer (OCIO) worked with officials from component agencies in order to revise its Capital Planning and Investment Control Directive and Instruction, and its Operational Analysis guidance. DHS plans to release its revised directive and guidance after DHS senior management approves DHS's updated System Engineering Life Cycle guide. Several component agency programs are already following the revised capital planning and operational analysis guidance and have submitted operational analyses as a part of their Exhibit 300 business cases.

    Recommendation: In order to improve DHS's policies and practices for ensuring that datamining systems used for counterterrorism are effective and provide necessary privacy protections, the Secretary of Homeland Security should direct the Chief Information Officer and Chief Procurement Officer to work with their counterparts at component agencies to identify steps to mitigate challenges related to the review and oversight of operational systems and to DHS's changing policy requirements and determine clear corrective actions, taking the impact on components and on individual program managers into account.

    Agency Affected: Department of Homeland Security

  3. Status: Closed - Implemented

    Comments: In order to provide for additional scrutiny of the privacy protections for the sensitive information systems that do not have publicly-available privacy impact assessments, DHS noted in its agency comments letter that the DHS Privacy Office planned to include an annex on unreleased privacy impact assessments in its Annual Report to Congress that was marked and handled with the appropriate national security (or other sensitive, but unclassified) restrictions. The agency further noted that members of Congress could request the relevant documents or schedule a briefing with the agency after reviewing the annex. Subsequently, the Privacy Office included this annex in its recent report to Congress. By completing this activity, the DHS Privacy Office can better assure it is maintaining appropriate transparency for its systems and of its actions.

    Recommendation: In order to improve DHS's policies and practices for ensuring that datamining systems used for counterterrorism are effective and provide necessary privacy protections, the Secretary of Homeland Security should direct the Chief Privacy Officer to develop requirements for providing additional scrutiny of privacy protections for the sensitive information systems that are not transparent to the public through privacy impact assessments (PIAs).

    Agency Affected: Department of Homeland Security

  4. Status: Closed - Implemented

    Comments: The DHS Privacy Office began a privacy compliance review of the ICEPIC program in September 2011. While the review was ongoing, the Chief Privacy Officer reviewed, approved, and in October 2011, published an updated privacy impact assessment of the program that included the Law Enforcement Information Sharing service. In the results of the privacy compliance review, published in December 2011, the office noted that by directing the expedited preparation and review of the PIA update, the DHS Privacy Office was able to bring the system into compliance with the E-Government Act and DHS policy (which obviated the need to shut down the program).

    Recommendation: In order to improve DHS's policies and practices for ensuring that datamining systems used for counterterrorism are effective and provide necessary privacy protections, the Secretary of Homeland Security should direct the Chief Privacy Officer to investigate whether the information sharing component of U.S. Immigration and Customs Enforcement (ICE) Pattern Analysis and Information Collection (ICEPIC), called the Law Enforcement Information Sharing Service, should be deactivated until a PIA that includes this component is approved.

    Agency Affected: Department of Homeland Security

  5. Status: Closed - Implemented

    Comments: Of the six programs we reviewed, two have taken steps to address shortfalls and resolve deficiencies, two have been retired, one was cancelled, and one is undergoing a reorganization that will result in a new concept of operations. Specifically, the Analytical Framework for Investment (AFI) and the Customs and Border Patrol's (CBP) TECS-Mod systems have passed multiple internal and external reviews, and resolved deficiencies. Both the Data Analysis and Research for Trade Transparency System (DARTTS)and the ICE Pattern Analysis and Information Collection (ICEPIC) systems have been retired. The Citizenship Immigration Data Repository (CIDR) effort was cancelled. CBP's Advanced Targeting System-Passenger (ATS-P) is in the process of a reorganization and the agency expects to complete a new concept of operations for the system in late fall 2015. This new concept of operations will likely trigger the need for new assessments of system effectiveness and privacy protections.

    Recommendation: In order to improve DHS's policies and practices for ensuring that datamining systems used for counterterrorism are effective and provide necessary privacy protections, the Secretary of Homeland Security should direct the the appropriate component agency administrators to ensure that the system program offices for Analytical Framework for Intelligence (AFI), Automated Targeting System (ATS)-Passenger (ATS-P), Citizen and Immigration Data Repository (CIDR), Data Analysis and Research for Trade Transparency System (DARTTS), ICEPIC, and TECS Modernization (TECS-Mod) address the shortfalls in evaluating system effectiveness and privacy protections identified in this report, including shortfalls in applying acquisition practices, ensuring executive review and approval, and consistently documenting executive reviews.

    Agency Affected: Department of Homeland Security

 

Explore the full database of GAO's Open Recommendations »

Jul 22, 2015

Jul 14, 2015

Jul 8, 2015

Jun 10, 2015

Jun 2, 2015

May 13, 2015

Looking for more? Browse all our products here