DHS Training:

Improved Documentation, Resource Tracking, and Performance Measurement Could Strengthen Efforts

GAO-14-688: Published: Sep 10, 2014. Publicly Released: Sep 10, 2014.

Additional Materials:

Contact:

Diana C. Maurer
(202) 512-9627
maurerd@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

What GAO Found

The Department of Homeland Security (DHS) has processes to evaluate training, track resources, and assess leader development. However, various actions could better position the department to maximize the impact of its training efforts.

Training evaluation: All five DHS components in GAO's review—U.S. Customs and Border Protection, U.S. Immigration and Customs Enforcement, the U.S. Coast Guard, the Transportation Security Administration, and the Federal Law Enforcement Training Center—have a documented process to evaluate their training programs. Their documented processes fully included three of six attributes of effective training evaluation processes identifying goals, programs to evaluate, and how results are to be used. However, the documented processes did not consistently include the other three attributes: methodology, timeframes, and roles and responsibilities (see table). By updating documentation to address these attributes, DHS components would have more complete information to guide its efforts in conducting effective evaluations.

Summary of Training Evaluation Attributes DHS Could Better Document

 

Customs and Border Protection

U.S. Coast Guard

Immigration and Customs Enforcement

Transportation Security Administration

Federal Law Enforcement Training Center

Methodology

Timeframes

Roles and responsibilities

: Documented evaluation processes fully included information to meet the attribute.

: Documented evaluation processes partly included some information to address a given attribute.

Source: GAO analysis of documented evaluation processes. I GAO-14-688

Capturing training cost: DHS identified efficiencies and cost savings for delivering a number of training programs. However, different methods are used for capturing training costs across the department, which poses challenges for reliably capturing these costs across DHS. Components capture training costs differently, contributing to inconsistencies in training costs captured across DHS. Variation in methods used to collect data can affect the reliability and quality of DHS-wide training program costs. However, DHS has not identified all challenges that contribute to these inconsistencies. DHS could improve its awareness about the costs of training programs DHS-wide and thereby enhance its resource stewardship by identifying existing challenges that prevent DHS from accurately capturing training costs and implementing corrective measures.

Leader development: DHS's Leader Development Program (LDP) Office is in the process of implementing a department-wide framework to build leadership skills. However, the LDP Office has not clearly identified program goals and the measures it uses to assess program effectiveness do not exhibit some attributes that GAO previously identified as key for successful performance measurement. These include linkage of performance measures to the program's goals, clarity, and establishment of measurable targets to assess the measures. By clearly identifying program goals and incorporating key attributes, the LDP could better ensure actionable information for identifying and making program improvements.

Why GAO Did This Study

DHS is the third-largest cabinet-level department in the federal government, with over 230,000 employees doing diverse jobs. To fulfill its complex mission, DHS's workforce must have the necessary skills and expertise. GAO previously reported on DHS's hiring and recruiting efforts. GAO was asked to assess DHS's training practices.

This report addresses (1) the extent to which DHS has documented processes to evaluate training and reliably capture costs and (2) the extent to which DHS measures the performance of its leader development programs. To conduct its work, GAO reviewed documented training evaluation processes, training cost data from fiscal year 2011 through fiscal year 2013, and leadership training programs. GAO also interviewed training officials at the department level and at the five DHS components selected for this review about their varieties of training and development programs. Information from these components cannot be generalized to all of DHS, but provides insights.

What GAO Recommends

GAO recommends that DHS update its documentation to fully reflect key attributes of an effective evaluation, identify challenges to and corrective measures for capturing training costs department-wide, and clearly identify LDP goals and ensure that LDP performance measures reflect key attributes. DHS concurred and identified actions to address our recommendations.

For more information, contact David C. Maurer at (202) 512-9627 or maurerd@gao.gov.

Recommendations for Executive Action

  1. Status: Open

    Comments: In September 2014, we reported on the Department of Homeland Security's (DHS) training efforts, including the extent to which DHS has a documented process to evaluate training and development programs. We found that all five DHS components in GAO's review--U.S. Customs and Border Protection, U.S. Immigration and Customs Enforcement, the U.S. Coast Guard, the Transportation Security Administration, and the Federal Law Enforcement Training Center--have a documented process to evaluate their training programs. Their documented processes fully included three of six attributes of effective training evaluation processes identifying goals, programs to evaluate, and how results are to be used. However, the documented processes did not consistently include the other three attributes: methodology, timeframes, and roles and responsibilities. We concluded that by updating documentation to address these attributes, DHS components would have more complete information to guide its efforts in conducting effective evaluations. We therefore recommended that DHS direct its components to ensure that their documented training evaluation processes fully address attributes for effective training evaluation processes as they are drafted, updated, or revised. As of July 2015, DHS reports that its Office of the Chief Human Capital Officer has included an action item in its draft Human Capital Development Plan to conduct a DHS-wide self-audit of training evaluation practices. This effort is tentatively to be initiated in April of 2016. Therefore, this recommendation remains open.

    Recommendation: To ensure effective evaluation of federal training programs and enhance DHS's stewardship of resources for federal training programs, the Secretary of Homeland Security should direct DHS components to ensure that their documented training evaluation processes fully address attributes for effective training evaluation processes as they are drafted, updated, or revised.

    Agency Affected: Department of Homeland Security

  2. Status: Open

    Comments: In September 2014, we reported on the Department of Homeland Security's (DHS) training efforts, including the extent to which DHS has a documented process to reliably capture costs. We found that DHS identified efficiencies and cost savings for delivering a number of training programs. However, different methods are used for capturing training costs across the department, which poses challenges for reliably capturing these costs across DHS. Components capture training costs differently, contributing to inconsistencies in training costs captured across DHS. Variation in methods used to collect data can affect the reliability and quality of DHS-wide training program costs. However, DHS has not identified all challenges that contribute to these inconsistencies. We concluded that DHS could improve its awareness about the costs of training programs DHS-wide and thereby enhance its resource stewardship by identifying existing challenges that prevent DHS from accurately capturing training costs and implementing corrective measures. We therefore recommended that DHS identify existing challenges that prevent DHS from accurately capturing training costs department-wide and, to the extent that the benefits of addressing those challenges exceed the costs, implement corrective measures to overcome these challenges. As of July 2015, DHS reports that its Office of the Chief Human Capital Officer and Office of the Chief Financial Officer are working together to research the issue and determine the best course of action to standardize training cost reporting. According to DHS officials, to date they have identified 111 accounting codes that may be suitable for tracking training costs, but suspect that some of definitions may need to be updated. DHS continues to work through the existing systems and coding to identify the best steps to get us to more accurate reporting of training costs. A date for completing efforts to address this recommendation has yet to be identified. Therefore, this recommendation remains open.

    Recommendation: To ensure effective evaluation of federal training programs and enhance DHS's stewardship of resources for federal training programs, the Secretary of Homeland Security should identify existing challenges that prevent DHS from accurately capturing training costs department-wide and, to the extent that the benefits of addressing those challenges exceed the costs, implement corrective measures to overcome these challenges..

    Agency Affected: Department of Homeland Security

  3. Status: Closed - Implemented

    Comments: In September 2014, we reported on the Department of Homeland Security's (DHS) training efforts, including the extent to which DHS measures the performance of its leader development programs. We found that DHS had established a Leader Development Program Office under the Office of the Chief Human Capital Officer to design, develop, and execute a department-wide leadership program. As of September 2014, the Leader Development Program Office was in the process of implementing a five-tier Leader Development Framework to build leadership skills across all staff levels, and had developed a program-wide assessment approach to analyze the impact of its efforts. However, we found that the Leader Development Program Office could strengthen its performance measurement approach. In particular, we found that the office had not clearly identified program goals, and that the 12 measures that the office developed to assess its performance did not consistently exhibit attributes we have previously identified as key for successful measurement. We recommended that DHS clearly identify Leader Development Program goals and ensure program performance measures include key attributes, such as linkage, clarity, and measurable targets. In December 2014, Leader Development Program Office officials provided us with updated documentation on the program's assessment approach. This documentation established ten program goals. It also explained how the program's performance measures link to the ten program goals and to department-wide goals. Further, the documentation established targets for each performance measure and provided clarification for ambiguous measures. These enhancements to the Leader Development Program assessment approach should help produce actionable information for the program's management to use in identifying the need for, and making, program improvements. Therefore, this recommendation is closed as implemented.

    Recommendation: In addition, to produce actionable information for DHS's LDP management to use in identifying the need for, and making, program improvements, the Secretary of Homeland Security should direct the Chief Human Capital Officer to clearly identify LDP goals and ensure LDP performance measures include key attributes, such as linkage, clarity, and measurable targets.

    Agency Affected: Department of Homeland Security

 

Explore the full database of GAO's Open Recommendations »

Jun 14, 2016

Jun 7, 2016

Jun 6, 2016

May 31, 2016

May 26, 2016

May 24, 2016

May 17, 2016

Apr 25, 2016

Looking for more? Browse all our products here