Workforce Investment Act:

Labor and States Have Taken Actions to Improve Data Quality, but Additional Steps Are Needed

GAO-06-82: Published: Nov 14, 2005. Publicly Released: Nov 14, 2005.

Additional Materials:

Contact:

George A. Scott
(202) 512-7003
contact@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

Federal programs carried out in partnership with states and localities continually balance the competing objectives of collecting uniform performance data with giving program implementers the flexibility they need. Our previous work identified limitations in the quality of performance data for the key employment and training program--the Workforce Investment Act (WIA). WIA relies on states and localities to work together to track and report on participant outcomes, and it changed the way outcomes are measured. Given the magnitude of changes and the impact such changes can have on data quality, we examined (1) the data quality issues that affected states' efforts to collect and report WIA performance data; (2) states' actions to address them; and (3) the actions the Department of Labor (Labor) is taking to address data quality issues, and the issues that remain.

Three key issues--flexibility in federal guidance, major changes to states' information technology (IT) systems, and limited monitoring--compromised states' early efforts to collect and report WIA performance data. Labor's initial guidance allowed states and local areas flexibility in deciding which jobseekers to track and when jobseekers leave services and get counted in the measures. As a result, states and local areas have differed on whom they track and for how long. States took various approaches to implement IT systems for meeting WIA reporting requirements. Thirty-nine states reported to us that they made major modifications to their IT systems since WIA was first implemented in 2000. Thirteen of them said the changes resulted in problems affecting data quality, and 5 states are still trying to resolve these problems. In addition, oversight of WIA performance data was insufficient at all levels during early implementation. Almost all states have made efforts to improve the quality of WIA performance data--at least 40 states have controls in their IT systems that capture WIA performance data, such as edit checks or exception reports to help screen for errors or missing data. Forty-three states have taken actions to clarify Labor's guidance and help local areas determine who should be tracked in the performance measures. In addition, most states said they monitor local areas by assessing local procedures and policies. Labor recently began addressing data quality issues, however, some issues remain. In 2004, Labor addressed some data quality concerns by requiring states to validate their data and ensure the accuracy of their performance outcomes. Most states told us that Labor's requirements have increased awareness of data quality at the state and local level. However, Labor does not have methods in place to review states' validation efforts or hold states accountable for complying with its requirements. Labor issued guidance requiring states to implement common performance measures on July 1, 2005, which clarified some key data elements, but does not address all the issues. Labor has some federal monitoring processes in place but lacks a standard monitoring guide to address data quality.

Recommendations for Executive Action

  1. Status: Closed - Implemented

    Comments: Labor originally planned to have standard error rates for report validation in place by program year 2004. However, Labor revised reporting requirements to implement a set of common measures across Employment and Training Administration programs. To allow states a sufficient transition period, Labor delayed establishing acceptable error rates and explored the possibility of including states' data validation results as part of the criteria for determining eligibility for incentive awards. In 2008, Labor reported that it included data validation results as one of the criteria for determining eligibility for WIA incentive awards (see TEGL 9-07). The results will be considered in determination of eligibility for awards made based on Program Year 2007 outcomes. In FY2010, Labor reported that it now factors the results of data validation into the determination of eligibility for WIA incentive grant awards. These results, which are determined by analysis of overall error rates, were used for the eligibility determination of the incentive grant awards based on Program Year 2008 (July 2008-June 2009) performance.

    Recommendation: To enhance the data validation requirements, the Secretary of Labor should ensure that steps are taken to hold states accountable to both the report validation and data element validation requirements.

    Agency Affected: Department of Labor

  2. Status: Closed - Implemented

    Comments: In 2006, Labor officials told us they planned to modify the data validation software to allow federal staff to pull a sub-sample of states' data validation records and conduct a federal review of these records. In 2008, Labor reported that it continues to be interested in making this enhancement to the software. However, due to resource constraints, priority has been given to enhancements to allow the states to continue using the software for data validation and to make it easier for the states to use in carrying out their data validation responsibilities. The regional staff continues to review state validation files and provide ongoing recommendations and suggestions to the states for improving their record-keeping. In FY2010, Labor reported that its regional office staff now conduct reviews of WIA participant files validated by states to ensure accuracy, and provide ongoing recommendations and suggestions to the states, helping them to improve their recordkeeping.

    Recommendation: To enhance the data validation requirements, the Secretary of Labor should conduct its own review of the WIA participant files validated by states to ensure that states did this correctly.

    Agency Affected: Department of Labor

  3. Status: Closed - Not Implemented

    Comments: In FY08, Labor stated that it provided guidance on who is considered a participant and counted in the performance measures in its Training and Employment Guidance Letter 17-05, Common Measures Policy for the Employment and Training Administration's (ETA) Performance Accountability System and Related Performance Issues. GAO believes, however that this revised policy guidance does not address the issue. Labor also noted that it provided technical assistance to states and local areas on performance-related issues, including participation. According to Labor, its Performance Enhancement Project (PEP) has been able to engage in a performance dialogue with state and local areas. PEP outreach and targeted technical assistance has taken many forms, such as webinars, conference calls, and face-to-face meetings. Labor adds that these and other technical support efforts (for example, the performance webinar series) have supported the communication of consistent and ongoing performance guidance. As needed, Labor used these opportunities to clarify policy. Technical assistance was also provided on how to best handle program flow and modify Management Information Systems to meet TEGL 17-05 requirements. In FY 2010, Labor reported that it continues to provide technical assistance to states on performance accountability and has convened a workgroup of state representatives to help further clarify the guidance on how to report self-service participants. While these steps may lead to improvements in performance data quality, Labor has not yet set a standard point of registration. GAO believes that actions taken do not fully address our recommendation.

    Recommendation: To address the inconsistencies in determining when participants should be registered and counted in the performance measures, the Secretary of Labor should determine a standard point of registration and monitor states to ensure that the policy is consistently applied.

    Agency Affected: Department of Labor

  4. Status: Closed - Implemented

    Comments: In 2006, Labor agreed with this recommendation and told us they were taking steps to develop a comprehensive monitoring guide for all regions that will include a section on the review of performance information. In 2008, Labor reported that it has used the Core Monitoring Guide since 2005, which has one chapter devoted to performance management, as its standard tool for monitoring performance-related subjects with our grantees. The agency also developed a draft data validation review guide as a supplemental tool that is being used to monitor the validity of reported data. This draft supplemental guide directs the reviewer to verify that state policy accurately reflects Labor's guidelines on program exiters, and to test a sample of participant files comparing the exit date with the last service date for validity. These reviews have been and continue to be conducted by each of Labor's regional offices. Labor also conducted 3 national training sessions for grantees in 2006 which included training on the provisions in TEGL 17-05 relating to the 90-day soft exit guidelines now in effect under the common measures. In FY2010, Labor reported that it is finalizing its data validation monitoring guide and expects to have the guide completed by fall 2010. The guide will supplement the overall Core Monitoring Guide.

    Recommendation: To address variations in federal monitoring practices, the Secretary of Labor should develop a standard comprehensive monitoring tool for WIA performance data that is used across all regions, including monitoring the new guidelines for determining when participants end services.

    Agency Affected: Department of Labor

 

Explore the full database of GAO's Open Recommendations »

Jul 23, 2014

May 15, 2014

Apr 9, 2014

Mar 5, 2014

Jan 23, 2014

Dec 20, 2013

Dec 6, 2013

Dec 2, 2013

Nov 4, 2013

Jul 19, 2013

Looking for more? Browse all our products here