Data Mining:

DHS Needs to Improve Executive Oversight of Systems Supporting Counterterrorism

GAO-11-742: Published: Sep 7, 2011. Publicly Released: Oct 7, 2011.

Additional Materials:

Contact:

David A. Powner
(202) 512-3000
contact@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

Data mining--a technique for extracting useful information from large volumes of data--is one type of analysis that the Department of Homeland Security (DHS) uses to help detect and prevent terrorist threats. While data-mining systems offer a number of promising benefits, their use also raises privacy concerns. GAO was asked to (1) assess DHS policies for evaluating the effectiveness and privacy protections of data-mining systems used for counterterrorism, (2) assess DHS agencies' efforts to evaluate the effectiveness and privacy protections of their data-mining systems, and (3) describe the challenges facing DHS in implementing an effective evaluation framework. To do so, GAO developed a systematic evaluation framework based on recommendations and best practices outlined by the National Research Council, industry practices, and prior GAO reports. GAO compared its evaluation framework to DHS's and three component agencies' policies and to six systems' practices, and interviewed agency officials about gaps in their evaluations and challenges..

As part of a systematic evaluation framework, agency policies should ensure organizational competence, evaluations of a system's effectiveness and privacy protections, executive review, and appropriate transparency throughout the system's life cycle. While DHS and three of its component agencies--U.S. Customs and Border Protection, U.S. Immigration and Customs Enforcement, and the U.S. Citizenship and Immigration Services--have established policies that address most of these key policy elements, the policies are not comprehensive. For example, DHS policies do not fully ensure executive review and transparency, and the component agencies' policies do not sufficiently require evaluating system effectiveness. DHS's Chief Information Officer reported that the agency is planning to improve its executive review process by conducting more intensive reviews of IT investments, including the data-mining systems reviewed in this report. Until such reforms are in place, DHS and its component agencies may not be able to ensure that critical data mining systems used in support of counterterrorism are both effective and that they protect personal privacy. Another aspect of a systematic evaluation framework involves ensuring that agencies implement sound practices for organizational competence, evaluations of a system's effectiveness and privacy protections, executive review, and appropriate transparency and oversight throughout a system's life cycle. Evaluations of six data mining systems from a mix of DHS component agencies showed that all six program offices took steps to evaluate their system's effectiveness and privacy protections. However, none performed all of the key activities associated with an effective evaluation framework. For example, four of the program offices executed most of the activities for evaluating program privacy impacts, but only one program office performed most of the activities related to obtaining executive review and approval. By not consistently performing necessary evaluations and reviews of these systems, DHS and its component agencies risk developing and acquiring systems that do not effectively support their agencies' missions and do not adequately ensure the protection of privacy-related information. DHS faces key challenges in implementing a framework to ensure systems are effective and provide privacy protections. These include reviewing and overseeing systems once they are in operation, stabilizing and implementing acquisition policies throughout the department, and ensuring that privacy-sensitive systems have timely and up-to-date privacy reviews. The shortfalls GAO noted in agency policies and practices provide insight into these challenges. Until DHS addresses these challenges, it will be limited in its ability to ensure that its systems have been adequately reviewed, are operating as intended, and are appropriately protecting individual privacy and assuring transparency to the public. GAO is recommending that DHS executives address gaps in agency evaluation policies and that component agency officials address shortfalls in their system evaluations. DHS concurred with GAO's recommendations and identified steps it is taking to address selected recommendations. The department also offered technical comments, which GAO incorporated as appropriate.

Recommendations for Executive Action

  1. Status: Open

    Comments: The Department of Homeland Security's Office of the Chief Information Officer (OCIO) established a policy in February 2014 that directs component chief information officers to ensure that DHS's components comply with Department-level IT policies and directives. In addition, DHS's Systems Engineering Life Cycle is in the process of being revised to ensure data quality assessments are included as part of system evaluations and OCIO is also currently in the process of identifying Component policy officials to being a formal policy review process. We will continue to monitor the agency's progress implementing the directive, issuing the updated Systems Engineering Life Cycle guide, and conducting reviews to address this recommendation.

    Recommendation: In order to improve DHS's policies and practices for ensuring that datamining systems used for counterterrorism are effective and provide necessary privacy protections, the Secretary of Homeland Security should direct the Chief Information Officer and Chief Procurement Officer to work with their counterparts at component agencies to ensure the consistency of component agencies' policies with DHS policies and proposed improvements to those policies, including requiring data quality assessments, requiring re-evaluations of operational systems, and establishing investment review boards with clearly defined structures for system review.

    Agency Affected: Department of Homeland Security

  2. Status: Open

    Comments: The Department of Homeland Security's Office of the Chief Information Officer (OCIO) stated that it has taken several steps to improve the review and oversight of operational systems. For example, the OCIO reported that it is in the process of reviewing comments on its Capital Planning and Investment Control Directive and Instruction and that its revised Operational Analysis guidance will be released once the Capital Planning and Investment Control Directive is approved. Once the guidance is approved and implemented, according to OCIO, component program managers will be required to submit operational analyses as part of their annual Exhibit 300 submissions. We will continue to monitor the department's actions in these areas.

    Recommendation: In order to improve DHS's policies and practices for ensuring that datamining systems used for counterterrorism are effective and provide necessary privacy protections, the Secretary of Homeland Security should direct the Chief Information Officer and Chief Procurement Officer to work with their counterparts at component agencies to identify steps to mitigate challenges related to the review and oversight of operational systems and to DHS's changing policy requirements and determine clear corrective actions, taking the impact on components and on individual program managers into account.

    Agency Affected: Department of Homeland Security

  3. Status: Closed - Implemented

    Comments: In order to provide for additional scrutiny of the privacy protections for the sensitive information systems that do not have publicly-available privacy impact assessments, DHS noted in its agency comments letter that the DHS Privacy Office planned to include an annex on unreleased privacy impact assessments in its Annual Report to Congress that was marked and handled with the appropriate national security (or other sensitive, but unclassified) restrictions. The agency further noted that members of Congress could request the relevant documents or schedule a briefing with the agency after reviewing the annex. Subsequently, the Privacy Office included this annex in its recent report to Congress. By completing this activity, the DHS Privacy Office can better assure it is maintaining appropriate transparency for its systems and of its actions.

    Recommendation: In order to improve DHS's policies and practices for ensuring that datamining systems used for counterterrorism are effective and provide necessary privacy protections, the Secretary of Homeland Security should direct the Chief Privacy Officer to develop requirements for providing additional scrutiny of privacy protections for the sensitive information systems that are not transparent to the public through privacy impact assessments (PIAs).

    Agency Affected: Department of Homeland Security

  4. Status: Closed - Implemented

    Comments: The DHS Privacy Office began a privacy compliance review of the ICEPIC program in September 2011. While the review was ongoing, the Chief Privacy Officer reviewed, approved, and in October 2011, published an updated privacy impact assessment of the program that included the Law Enforcement Information Sharing service. In the results of the privacy compliance review, published in December 2011, the office noted that by directing the expedited preparation and review of the PIA update, the DHS Privacy Office was able to bring the system into compliance with the E-Government Act and DHS policy (which obviated the need to shut down the program).

    Recommendation: In order to improve DHS's policies and practices for ensuring that datamining systems used for counterterrorism are effective and provide necessary privacy protections, the Secretary of Homeland Security should direct the Chief Privacy Officer to investigate whether the information sharing component of U.S. Immigration and Customs Enforcement (ICE) Pattern Analysis and Information Collection (ICEPIC), called the Law Enforcement Information Sharing Service, should be deactivated until a PIA that includes this component is approved.

    Agency Affected: Department of Homeland Security

  5. Status: Open

    Comments: We received updates on the systems we reviewed at the Department of Homeland Security's (DHS) component agencies. While all have made progress in completing activities to address our recommendation, there is still significant work that remains. We will continue to monitor DHS's activities in this area.

    Recommendation: In order to improve DHS's policies and practices for ensuring that datamining systems used for counterterrorism are effective and provide necessary privacy protections, the Secretary of Homeland Security should direct the the appropriate component agency administrators to ensure that the system program offices for Analytical Framework for Intelligence (AFI), Automated Targeting System (ATS)-Passenger (ATS-P), Citizen and Immigration Data Repository (CIDR), Data Analysis and Research for Trade Transparency System (DARTTS), ICEPIC, and TECS Modernization (TECS-Mod) address the shortfalls in evaluating system effectiveness and privacy protections identified in this report, including shortfalls in applying acquisition practices, ensuring executive review and approval, and consistently documenting executive reviews.

    Agency Affected: Department of Homeland Security

 

Explore the full database of GAO's Open Recommendations »

Oct 14, 2014

Sep 30, 2014

Sep 24, 2014

Sep 18, 2014

Sep 17, 2014

Sep 10, 2014

Sep 9, 2014

Sep 8, 2014

Looking for more? Browse all our products here