This is the accessible text file for GAO report number GAO-14-4 
entitled 'Workforce Investment Act: DOL Should Do More to Improve the 
Quality of Participant Data' which was released on December 20, 2013. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as 
part of a longer term project to improve GAO products' accessibility. 
Every attempt has been made to maintain the structural and data 
integrity of the original printed product. Accessibility features, 
such as text descriptions of tables, consecutively numbered footnotes 
placed at the end of the file, and the text of agency comment letters, 
are provided but may not exactly duplicate the presentation or format 
of the printed version. The portable document format (PDF) file is an 
exact electronic replica of the printed version. We welcome your 
feedback. Please E-mail your comments regarding the contents or 
accessibility features of this document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

United States Government Accountability Office: 
GAO: 

Report to Congressional Requesters: 

December 2013: 

Workforce Investment Act: 

DOL Should Do More to Improve the Quality of Participant Data: 

GAO-14-4: 

GAO Highlights: 

Highlights of GAO-14-4, a report to congressional requesters. 

Why GAO Did This Study: 

Having reliable program data is important in effectively managing a 
program. However, there have been longstanding concerns about the 
quality of data on job seekers enrolled in the WIA Adult and 
Dislocated Worker Programs, which rely on states and local areas to 
track participants and the services they receive. Given these concerns 
and WIA’s anticipated reauthorization, GAO was asked to examine the 
data on these WIA participants. This report addresses: (1) the factors 
that have affected the ability to report consistent and complete data 
on participants in the WIA Adult and Dislocated Worker Programs, and 
(2) actions that DOL has taken to improve the quality of these data. 
To conduct this work, GAO reviewed relevant federal laws, regulations, 
guidance, and documentation from DOL. GAO interviewed officials from DOL
’s national and regional offices and state and local workforce 
officials from a nongeneralizable sample of eight states. GAO also 
analyzed WIA data from program year 2011 to determine the number of, 
characteristics of, and services provided to WIA participants. 

What GAO Found: 

Flexibility in the Department of Labor’s (DOL) data reporting guidance 
and limitations in some state information systems continue to impair 
the quality of the data on participants in the Workforce Investment 
Act (WIA) Adult and Dislocated Worker Programs. The flexibility in the 
guidance stems from the inherent nature of WIA, which allows states 
and local areas to tailor program design and service delivery to their 
needs. As a result, DOL’s guidance on collecting and reporting the 
data allows variation in how some WIA data elements are defined, 
collected, and reported. Specifically, an American Job Center—formerly 
known as a one-stop center—can choose to provide certain basic 
services exclusively through WIA programs, exclusively through a 
partner program, or through a blend of both WIA and partner programs. 
However, this flexibility involves variations in data reporting that 
have contributed to inconsistencies among states regarding when job 
seekers are counted as WIA participants. Moreover, some aspects of 
DOL’s guidance are open to interpretation, leaving it to states to 
define variables such as type of training service received, further 
contributing to data inconsistencies. In addition, some state 
information systems used to collect and report WIA participant data 
have limitations that hamper the affected states’ ability to report 
uniform and complete data. For example, data are incomplete to the 
extent that states may not have information systems that can track 
participants who access services online without significant staff 
assistance. Having inconsistent and incomplete data makes it difficult 
for DOL to compare data on program participants across states or to 
aggregate the data at a national level. 

DOL engages in various oversight activities designed to ensure the 
accuracy of states’ data on participants in the WIA Adult and 
Dislocated Worker Programs and has taken steps to improve data 
consistency across states. However, DOL does not consistently use the 
results of its oversight to identify and resolve systemic data issues 
nor has it evaluated the effect of oversight on the quality of WIA 
participant data. Specifically, DOL requires states to validate the 
data they collect and report on participants in the WIA Adult and 
Dislocated Worker Programs on an annual basis, but it does not 
strategically use the findings from this effort to identify systemic 
data issues or improve the quality of the data. Similarly, although 
DOL’s regional offices review a sample of each state’s WIA participant 
files every few years to assess states’ compliance with data reporting 
and validation requirements, DOL officials said they have not analyzed 
the findings from the most recent reviews to identify nation-wide 
reporting issues. DOL has taken steps to improve the consistency of 
the data by providing general technical assistance to states and local 
areas and through standardizing the way DOL collects WIA data. For 
example, since 2007, two states have been piloting a unified reporting 
system developed by DOL that uses standardized data definitions and is 
integrated across certain American Job Center programs administered by 
DOL. However, DOL officials said they have no plans to evaluate the 
system before expanding it to other states. Without an evaluation, DOL 
does not know what impact the pilot has had on the quality of WIA 
participant data. 

What GAO Recommends: 

GAO recommends that DOL take steps to improve the consistency and 
completeness of data reported across states and to promote a 
continuous process for improving the data’s quality. DOL officials did 
not agree or disagree with GAO’s overall recommendations and detailed 
how data quality is being addressed primarily through existing 
efforts. However, GAO believes that the recommendations remain valid 
as discussed in the report. 

View [hyperlink, http://www.gao.gov/products/GAO-14-4]. For more 
information, contact Revae E. Moran at (202) 512-7215 or 
moranr@gao.gov. 

[End of section] 

Contents: 

Letter: 

Background: 

DOL's Guidance and Limitations in State Information Systems Have 
Resulted in Inconsistent and Incomplete National Data on WIA 
Participants: 

DOL Has Increased Its Oversight Efforts and Taken Steps to Improve 
Data Consistency, but Data Quality Issues Have Not Been Resolved: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments and Our Evaluation: 

Appendix I: Scope and Methodology: 

Appendix II: Data on Participants in the WIA Adult and Dislocated 
Worker Programs for Program Year 2011: 

Appendix III: Comments from the Department of Labor: 

Appendix IV: GAO Contact and Staff Acknowledgments: 

Related GAO Products: 

Tables: 

Table 1: American Job Center Network Mandatory Partner Programs and 
Related Federal Agencies: 

Table 2: Most Prevalent Issues Identified During DOL's Regional 
Reviews of States' Case Files, Program Years 2006-2011A: 

Table 3: Selected Site Visit States and Local Areas: 

Figures: 

Figure 1: Sequence of Services for the WIA Adult and Dislocated Worker 
Programs: 

Figure 2: Flow of WIA Data from Local Area to State to DOL: 

Figure 3: The Impact of Two States' Service Delivery Models on When 
They Typically Count Job Seekers As WIA Adult Program Participants: 

Figure 4: Total Participants Enrolled in the Adult and Dislocated 
Worker Programs, Program Year 2011: 

Figure 5: Highest Level of Service Received by Adult and Dislocated 
Worker Program Participants, Program Year 2011: 

Figure 6: Duration of Training for Adult and Dislocated Worker Program 
Participants Who Completed or Withdrew from Training, Program Year 
2011: 

Figure 7: Type of Training Services Provided to Adult and Dislocated 
Worker Program Participants, Program Year 2011: 

Figure 8: Percentage of Adult and Dislocated Worker Program 
Participants Registered in Selected Partner Programs, Program Year 
2011: 

Figure 9: Adult and Dislocated Worker Program Participant 
Demographics, Program Year 2011: 

Abbreviations: 

DOL: The Department of Labor: 

ETA: Employment and Training Administration: 

SPRA: Social Policy Research Associates: 

WIA: The Workforce Investment Act of 1998: 

WIASRD: Workforce Investment Act Standardized Record Data: 

WISPR: Workforce Investment Streamlined Performance Reporting System: 

[End of section] 

GAO:
United States Government Accountability Office: 
441 G St. N.W. 
Washington, DC 20548: 

December 2, 2013: 

The Honorable John Kline: 
Chairman: 
Committee on Education and the Workforce: 
House of Representatives: 

The Honorable Virginia Foxx: 
Chairwoman: 
Subcommittee on Higher Education and Workforce Training: 
Committee on Education and the Workforce: 
House of Representatives: 

The Honorable Joe Heck: 
House of Representatives: 

The Honorable Howard P. McKeon: 
House of Representatives: 

Having reliable program data is an important factor in being able to 
effectively manage and evaluate a program. Our previous work on the 
Workforce Investment Act of 1998 (WIA), however, has raised questions 
about the quality of data that states report to the U.S. Department of 
Labor (DOL) on the number of individuals they serve.[Footnote 1] WIA, 
the centerpiece of the nation's employment and training system, 
established three separately funded programs--Adult, Dislocated 
Worker, and Youth--and created a comprehensive one-stop system, now 
known as the American Job Center network, for the delivery of many 
federally funded employment and training program services.[Footnote 2] 
At this writing, the Congress is considering proposals to reauthorize 
WIA, which has been due for reauthorization since 2003.[Footnote 3] 

In our previous work, we found that the federal government's efforts 
to collect and report accurate, consistent WIA program performance 
data from states were affected by (1) flexibility in federal guidance 
on collecting and reporting the data, (2) major changes to states' 
information systems that resulted in lost data and other issues, and 
(3) limited monitoring by the Department of Labor (DOL). We also found 
that DOL's WIA data do not include information on all participants in 
the Adult and Dislocated Worker Programs and that the data were not 
comparable across states and local areas.[Footnote 4] Similarly, DOL's 
Office of Inspector General found that states were reporting 
inconsistent data on WIA participants in the Adult and Dislocated 
Worker Programs because DOL's guidance did not clarify how certain 
participants should be counted.[Footnote 5] Given the longstanding 
concerns about the quality of the data and in anticipation of WIA's 
reauthorization, you asked us to examine the data on WIA participants. 
This report addresses the following questions: 

1. What factors have affected the ability to report consistent and 
complete data on participants in the WIA Adult and Dislocated Worker 
Programs, and: 

2. What actions has DOL taken to improve the quality of participant 
data? 

To address our objectives, we reviewed relevant federal laws and 
regulations, as well as the guidance DOL provides to states for 
collecting and reporting data on participants in the WIA Adult and 
Dislocated Worker Programs.[Footnote 6] We also interviewed officials 
from DOL's Office of Inspector General and representatives of two 
national workforce-related organizations. To better understand the 
challenges faced by states in reporting data on these WIA programs, we 
visited or telephoned DOL's Employment and Training Administration's 
(ETA) six regional offices and a nongeneralizable sample of eight 
states. Within each state, we visited or contacted at least one 
comprehensive American Job Center--formerly known as a one-stop center-
-or a local workforce board. We selected the states to provide 
diversity on the basis of: (1) geographic location, (2) total federal 
spending on the Adult and Dislocated Worker Programs in program year 
2010, (3) the extent of data issues identified by DOL's data 
contractor in the fourth quarter of program year 2010, (4) whether the 
state reported participants who received only core "self-service," and 
(5) the number of local areas within the state.[Footnote 7] 

To determine the actions DOL has taken to improve the quality of 
participant data for the WIA Adult and Dislocated Worker Programs, we 
interviewed officials from ETA's national office and reviewed 
documentation and reports related to DOL's various initiatives. We 
also assessed the reliability of program year 2011 data on 
participants in the Adult or Dislocated Worker Program from DOL's 
Workforce Investment Act Standardized Record Data (WIASRD) database by 
testing the data electronically and interviewing knowledgeable agency 
officials and DOL's data contractor about the actions they take to 
validate the data.[Footnote 8] We found the data to be sufficiently 
reliable for the purposes of providing estimates for the number of, 
characteristics of, and services provided to participants whose 
information is recorded by DOL as having been served by the WIA Adult 
or Dislocated Worker Program.[Footnote 9] However, we did not find 
those data to be reliable for other purposes such as making state-to-
state comparisons because of variations in how states collect and 
report participant data for the WIA Adult and Dislocated Worker 
Programs. See appendix I for additional information on our scope and 
methodology. 

We conducted this performance audit from August 2012 through November 
2013 in accordance with generally accepted auditing standards. Those 
standards require that we plan and perform the audit work to obtain 
sufficient, appropriate evidence to provide a reasonable basis for our 
findings and conclusions based on our audit objectives. We believe 
that the evidence obtained provides a reasonable basis for our 
findings and conclusions based on our audit objectives. 

Background: 

WIA requires states and local areas to bring together a number of 
federally funded employment and training programs into a comprehensive 
workforce investment system, the American Job Center network. These 
programs--including the Adult and Dislocated Worker Programs--are 
known as mandatory partners, and must provide services through this 
network (see table 1). 

Table 1: American Job Center Network Mandatory Partner Programs and 
Related Federal Agencies[A]: 

Federal Agency: Department of Labor; 
Mandatory Partner Programs: 
WIA Adult; 
WIA Dislocated Worker; 
WIA Youth; 
Employment Service (Wagner-Peyser); 
Trade Adjustment Assistance program; 
Veterans' Employment and Training programs; 
Unemployment Insurance; 
Job Corps; 
Senior Community Service Employment Program; 
Employment and Training for Migrant and Seasonal Farm Workers; 
Training for Native Americans; 
YouthBuild. 

Federal Agency: Department of Education; 
Mandatory Partner Programs: 
Postsecondary Vocational Rehabilitation Program; 
Adult Education and Literacy; 
Vocational Education (Perkins Act). 

Federal Agency: Department of Health and Human Services; 
Mandatory Partner Programs: 
Community Services Block Grant. 

Source: Pub. L. No. 105-220 and U.S. Department of Labor. 

[A] WIA specified 18 required partner programs administered by four 
federal agencies. One of these, the Welfare to Work grant program, 
administered by the Department of Labor, was discontinued in 2004. 
Employment and training activities administered by the Department of 
Housing and Urban Development (HUD) were also specified as a required 
partner but HUD does not currently administer any training and 
employment programs. Furthermore, in 2006, the YouthBuild program was 
transferred to the Department of Labor from the Department of Housing 
and Urban Development. 

[End of table] 

The WIA Adult and Dislocated Worker Programs are designed to provide 
quality employment and training services to assist eligible 
individuals to find and qualify for employment and to help employers 
find the skilled workers they need. The Adult Program provides 
services to individuals over the age of 18 who are job seekers, 
although states and local areas must give priority of service to low-
income individuals if funds are determined to be limited.[Footnote 10] 
The Dislocated Worker Program provides services to workers who have 
been or will be terminated or laid off from employment.[Footnote 11] 
For fiscal year 2013, Congress appropriated over $1.9 billion for the 
Adult and Dislocated Worker Programs: $730 million for the Adult 
Program and $1.2 billion for the Dislocated Worker Program.[Footnote 
12] 

DOL's Employment and Training Administration administers the WIA Adult 
and Dislocated Worker Programs and oversees their implementation, 
which is carried out by states and local areas. Each state must have 
one or more designated local workforce investment area, and each local 
area must have at least one comprehensive American Job Center where 
job seekers can receive core services and access other programs and 
activities offered by the mandatory partners.[Footnote 13] Although 
each local area must have one comprehensive center, under WIA, the 
mandatory partners have flexibility in the way they provide services 
through the American Job Center network, and can co-locate services on 
site or make referrals to external service providers or training, 
including to local community colleges. 

WIA provides three tiers, or levels, of service for adult and 
dislocated workers: (1) core, (2) intensive, and (3) training. 
[Footnote 14] 

* Core services include basic services, such as job search or résumé-
building assistance, and may be accessed with or without staff 
assistance. 

* Intensive services include activities such as staff-assisted 
comprehensive assessment of a participant's skill levels and case 
management. 

* Training services include activities such as occupational skills or 
on-the-job training (see figure 1). 

Service at one level and a determination that a participant is unable 
to obtain employment through that service are prerequisites for 
service at the next level, although WIA does not specify the amount of 
time an individual must spend or the number of attempts that must be 
made to gain employment before moving to the next level.[Footnote 15] 
Job seekers who receive only core services that are self-service and 
informational in nature are not counted in the programs' performance 
measures, but DOL requires that states count such individuals as 
program participants.[Footnote 16] Self-service and informational 
activities can be accessed either at an American Job Center or 
remotely, such as when a job seeker searches for employment over an 
internet connection to an American Job Center from a home computer. 

Figure 1: Sequence of Services for the WIA Adult and Dislocated Worker 
Programs: 

[Refer to PDF for image: illustrated list] 

Core services: 
Basic services such as: 
* Orientation to American Job Center; 
* Job search and placement assistance; 
* Labor market information. 

Intensive services: 
Services that require greater staff involvement such as:
* Comprehensive assessment; 
* Development of individual employment plans; 
* Case management; 
* Prevocational services such as developing interview skills. 

Training services: 
Types of training services include: 
* On-the-job training; 
* Skills upgrading and retraining; 
* Entrepreneurial training; 
* Adult education or English as a second language in conjunction with 
other training; 
* Customized training; 
* Other occupational skills training. 

Source: GAO analysis of statute and Department of Labor guidance. 

[End of figure] 

As part of its oversight, DOL collects program data from states, which 
are used to assess how well the Adult and Dislocated Worker Programs 
are working. States must submit quarterly and annual performance 
reports to DOL, in addition to uploading individual records on a 
quarterly basis to DOL's national WIASRD database. Specifically, 
states must submit quarterly and supplemental monthly performance 
reports, a validated annual performance report at the end of each 
year, and quarterly WIASRD files for each reporting quarter for a 
program year.[Footnote 17] The WIASRD files include demographic and 
characteristic information for participants in the WIA Adult and 
Dislocated Worker Programs and information about services received 
through these WIA programs, as well as through some other partner 
programs.[Footnote 18] The process of collecting and reporting WIA 
data involves all three levels of government. More specifically, 
participant data are typically collected by staff at American Job 
Centers and entered into a state or local information system. In some 
states, local staff may enter data directly into a statewide 
information system; in other states, local areas may use their own 
individualized information systems to enter data from which they then 
must extract and compile for submission to the state. After the data 
is submitted to the state agency, it is compiled and formatted for the 
various submissions to DOL, including for quarterly WIASRD record 
layout submissions. (see figure 2). 

Figure 2: Flow of WIA Data from Local Area to State to DOL: 

[Refer to PDF for image: illustration] 

Local level: 
1. Local data collection and data entry: 
Local staff collects data from participants: 
* Forms
* Copies of support documentation; 
Stored in local or state information system. 

State level: 
2. State information system: 
State data extracted for federal reporting: 
* Quarterly/annual aggregate reports; 
* Quarterly WIASRD files. 

Federal level: 
3. Federal information system and reporting: 
* Quarterly/annual aggregate reports; 
* Data contractor: WIASRD National Report. 

Source: GAO analysis and Art Explosion (images). 

[End of figure] 

DOL's Guidance and Limitations in State Information Systems Have 
Resulted in Inconsistent and Incomplete National Data on WIA 
Participants: 

Flexibility in DOL's Guidance Continues to Affect the Consistency of 
Data: 

DOL's guidance to states, in the form of Training and Employment 
Guidance Letters and Training and Employment Notices, details how 
states should collect and report data on participants in the WIA Adult 
and Dislocated Worker Programs. However, the flexibility in the 
guidance DOL provides to states makes it difficult for DOL to provide 
consistent national data on participants in these programs. The 
flexibility in the guidance stems from the flexibility inherent in 
WIA, which allows states and local areas to tailor service delivery to 
their needs.[Footnote 19] Consequently, DOL's guidance is designed to 
accommodate the different ways that states and local areas can deliver 
services through the American Job Center network, including decisions 
about whether to enroll participants in partner programs. This 
flexibility, however, results in variations in how states and local 
areas report participant data, which makes it challenging for DOL to 
aggregate WIA data at the national level. In particular, it has 
created inconsistencies among states in when a job seeker is counted 
as a participant in the WIA Adult or Dislocated Worker Program. In 
addition, DOL's guidance is open to interpretation, allowing states to 
define and report some variables differently, further contributing to 
inconsistencies in the data that states report to DOL on these 
programs. For example, while DOL requires states to report the type of 
training service provided to WIA participants who receive such 
services, the agency's guidance only lists the six broad categories of 
training services states must report without defining or describing 
them in detail. In accordance with government internal control 
standards, management is responsible for developing policies and 
procedures to achieve program objectives and clearly communicating 
these policies and procedures to facilitate understanding and 
consistent implementation.[Footnote 20] 

Flexibility in When to Count Participants under WIA: 

The flexibility in DOL's guidance on when states and local areas 
should count individuals as participants in the WIA Adult and 
Dislocated Worker Programs has resulted in inconsistencies in the data 
reported on program participants. WIA allows for flexibility in the 
extent to which Adult and Dislocated Worker Program services are 
integrated with those from other programs.[Footnote 21] DOL's guidance 
encourages, but does not require, states to integrate WIA services 
with those from other partner programs so that job seekers have access 
to a coordinated system of employment and training services.[Footnote 
22] As a result, states' service delivery models differ in the extent 
of integration between WIA services and other programs. For example, 
state officials that we interviewed told us that they funded core 
services at their American Job Centers exclusively through WIA, 
exclusively through a partner program, or through a blend of both WIA 
and partner program funds. According to officials from DOL's national 
office, states and local areas are best positioned to determine the 
mix of services that will meet the needs of their job seekers. 
However, depending on which service delivery model a state or local 
area selects, job seekers receiving the same types of services may be 
counted as WIA participants at different points in time, or they may 
never be counted under the WIA Adult or Dislocated Worker Program. 

This variability in when states count job seekers under WIA has 
resulted in a lack of consistency and considerable differences in 
whether states report job seekers at American Job Centers as 
participants in the WIA Adult Program, the WIA Dislocated Worker 
Program, or both. More specifically, some states with relatively high 
general populations are counting fewer job seekers in these programs 
than states with much lower populations.[Footnote 23] For example, 
according to the state officials we interviewed, Utah's American Job 
Centers integrate WIA Adult Program funding with funding for the 
Wagner-Peyser Program to provide core services. As a result, every job 
seeker aged 18 years or older who receives core services in Utah is 
counted as a participant in both programs, as permitted under DOL's 
guidance.[Footnote 24] In comparison, state officials in California 
told us that the state typically funds core services exclusively 
through the Wagner-Peyser Program and therefore counts all individuals 
accessing core services from an American Job Center as participants in 
the Wagner-Peyser Program, but not as participants in the WIA Adult 
Program or the Dislocated Worker Program. Therefore, only job seekers 
who meet the eligibility criteria for the WIA Adult or Dislocated 
Worker Program and receive intensive or training services funded by 
those programs are counted as WIA participants. Because of this 
variability, the total counts of WIA Adult Program participants and 
WIA Dislocated Worker Program participants represent different 
populations of job seekers in different states, depending on the 
service delivery model the state uses. 

As a result of these differences, Utah ranked 3rd out of 53 states in 
the total number of participants it served in the WIA Adult Program in 
program year 2011, even though it ranked 36th in overall population. 
[Footnote 25] At the same time, California ranked 27th in the total 
number of participants served in the WIA Adult Program, even though it 
was the most populous state (see figure 3). 

Figure 3: The Impact of Two States' Service Delivery Models on When 
They Typically Count Job Seekers As WIA Adult Program Participants: 

[Refer to PDF for image: illustrated table] 

State: Utah; 
Counts job seekers as WIA participants when they access core services
funded by WIA or Wagner-Peyser: 
Core services: Yes; 
Intensive and training services: Yes; 
Counted WIA participants: 14. 

State: California; 
Counts job seekers as WIA participants only if they require intensive 
or training services funded by WIA; 
Core services: No; 
Intensive and training services: Yes; 
Counted WIA participants: 2. 

Source: GAO analysis of Department of Labor participant data. 

[End of figure] 

The extent to which states and local areas comply with DOL's reporting 
guidelines also affects the consistency of data on program 
participants. We visited three American Job Centers that provided core 
services to job seekers funded by WIA without counting these job 
seekers as WIA participants despite DOL guidance that these 
individuals be counted under WIA. For example, at two American Job 
Centers in Georgia, officials stated that they were providing core 
services funded by WIA to job seekers without enrolling these 
individuals in any WIA program. Officials at one American Job Center 
in Georgia added that these individuals are not counted, in part, 
because it is burdensome to collect participant data on jobseekers who 
receive only core services. We noted a similar compliance issue at an 
American Job Center in Illinois, and DOL's case file reviews indicate 
that other local areas may be providing WIA-funded services to 
individuals without recording them as participants.[Footnote 26] As a 
result, cross-state comparisons of the number of participants in WIA 
programs are potentially misleading and DOL is limited in the extent 
to which it can use national WIA participant data to manage these 
programs. In addition, without accurate information on program 
participants, DOL and lawmakers run the risk of making decisions about 
WIA policy and funding that are not based on a true picture of the 
number of participants and the program's outcomes. 

Flexibility in Reporting on Services Provided by Partner Programs: 

Participation Date: Flexibility in DOL's guidance contributes to 
differences in how states collect and report WIA data on services 
provided by partner programs. For example, DOL's guidance allows 
states and local areas flexibility in what date to report as the 
participation date for WIA programs.[Footnote 27] More specifically, 
for WIA participants who receive services funded by a program other 
than WIA, states and local areas can record the participation date in 
WIASRD as either the first date an individual received WIA services or 
the first date an individual received services from any other partner 
program. If, for example, a participant received services from the 
Wagner-Peyser Program on April 1st and then received services from WIA 
on December 15th, states can record the date of WIA program 
participation as either April 1st or December 15th. Among the state 
officials we interviewed in the eight states we selected, five of them 
said their states recorded the participation date as the first date a 
WIA service was provided to a job seeker and three of them said their 
states recorded the participation date as the first date a partner 
program's service was provided to a job seeker. This flexibility 
results in inconsistent reporting of the participation date, which 
makes it difficult for DOL to compare the length of WIA program 
participation across states. 

Service Dates: Similarly, DOL's guidance allows states and local areas 
flexibility in how they collect and report data on the services WIA 
participants receive from partner programs. This makes it difficult to 
make state-to-state comparisons of the types of services participants 
in the WIA Adult or Dislocated Worker Programs receive, when they 
receive them, and whether they are funded by WIA or a partner program. 
For example, when a staff-assisted core service is provided by WIA, 
DOL's guidance instructs states and local areas to report the date a 
participant received the service for the first time. However, when 
another partner program provides the service, officials from DOL's 
national office said that they encourage but do not require states to 
report the first date that staff-assisted core services were provided. 
[Footnote 28] As a result, some states used WIASRD to report core 
services funded by partner programs, while others did not. Our 
analysis of program year 2011 WIASRD data showed that 11 states had 
missing data for "Date of First Staff-Assisted Core Service" for over 
half of the participants who received intensive or training services 
from either the WIA Adult Program or the WIA Dislocated Worker 
Program.[Footnote 29] 

For these states with missing data, DOL's contractor uses the 
participation date in WIA to estimate when a participant received 
his/her first staff-assisted core service.[Footnote 30] However, to 
the extent that states purposely leave the "Date of First Staff-
Assisted Core Service" blank because the service was provided by a 
partner program, this revision artificially inflates the number of WIA 
participants who are counted as having received these services under 
WIA. This also makes it difficult for DOL to know whether the services 
reported in WIASRD were funded by WIA or a partner program. 

Broad Definitions for Selected Variables: 

DOL's guidance to states for some WIASRD variables is open to 
interpretation, resulting in differences in how states define or 
report them, further contributing to the challenges DOL faces in 
reporting consistent national data on participants in these programs. 
According to officials from DOL's national office, the guidance on how 
to report select variables in WIASRD is intentionally broad to 
accommodate the different ways states deliver services. Some of the 
state and local officials we interviewed told us that the definitions 
for the following variables were relatively broad and open to 
interpretation: 

"Type of training": While DOL requires states to report the type of 
training service provided to all WIA participants who receive such 
services, the agency's guidance only lists the six broad categories of 
training states must report.[Footnote 31] Officials from one regional 
office and one state said that interpretations of these categories can 
vary across states and local areas, which can define the categories 
differently, particularly the one for "Other Occupational Skills 
Training." According to our analysis of program year 2011 WIASRD data, 
most states reported that at least 75 percent of their training 
recipients in both the WIA Adult Program and the Dislocated Worker 
Program received "Other Occupational Skills Training," a broad 
category with no clear definition or description. While this is a 
legitimate and appropriate type of training, such as for job seekers 
with Individual Training Accounts allowed under WIA, it is unclear 
what comprises this category of training, which is attributed to over 
two-thirds of participants in training, because states are not 
required to elaborate on it. Without clearly defined categories, 
policymakers and program managers do not have relevant information to 
use to compare or evaluate the effectiveness of the different types of 
training services provided to WIA participants. Officials from DOL's 
national office acknowledged that the categories were not clearly 
defined but noted the trade-off between providing flexibility to 
states and local areas and being too prescriptive. 

"Core Services": According to DOL's guidance, core services that 
require "significant staff involvement" are considered to be staff-
assisted core services in contrast to self-services and informational 
activities defined as "core and informational services that do not 
require significant staff involvement."[Footnote 32] While DOL has 
provided detailed written guidance to states to distinguish staff-
assisted services from self-services, the guidance remains open to 
interpretation and requires local area staff to make subtle and 
subjective distinctions concerning the level of staff assistance 
provided. Officials from three states and one region told us that it 
was difficult to differentiate staff-assisted services from self-
services. As a result, states' definitions of staff-assisted core 
services may vary, which could affect how many participants are 
included in WIA's performance measures. 

Officials from three states also told us that some of DOL's reporting 
guidance is open to interpretation. This can create challenges for 
states to communicate a uniform interpretation to local areas. For 
example, officials from Massachusetts said that, because each American 
Job Center has some degree of flexibility in the services they offer 
and how they are delivered, interpretation of DOL guidance may vary by 
local area. 

Limitations in State Information Systems Hamper DOL's Ability to 
Collect and Report Uniform and Complete National Data: 

According to officials from five regional offices, state information 
systems vary in quality, and some lack the capability to collect and 
report all of the required data, which has affected the comparability 
and completeness of some of the data on participants in the WIA Adult 
and Dislocated Worker Programs. WIA requires each state to build and 
manage its own information system for collecting and reporting data on 
WIA participants.[Footnote 33] Some states have developed their own 
unique information systems while others, including three of the eight 
states whose officials we interviewed, contract with companies to 
develop and manage their WIA data systems.[Footnote 34] However, some 
WIASRD variables may not be reported due to limitations in some state 
information systems. As a result, some of the data on participants are 
inconsistent and incomplete, which makes it difficult for DOL to 
compare data on program participants across states or to aggregate the 
data at a national level. In addition, without having accurate data on 
the number and types of participants, it is difficult for DOL and 
lawmakers to make sound policy and funding decisions about these 
programs. 

For example, according to officials from DOL's national office and 
from three of its regional offices, some state systems cannot account 
for participants in the WIA Adult Program who receive only core self-
services, which affects the completeness of the data.[Footnote 35] In 
addition, DOL's guidance notes that the use of a voluntary online 
registration process to collect information on these self-service-only 
participants is the best approach for improving consistency in the 
reporting of these participants.[Footnote 36] Nevertheless, some state 
information systems are not capable of reporting data on participants 
who access services online, according to officials from one regional 
office. In 2010, DOL's reporting guidance acknowledged the variability 
in reporting on self-service-only participants with some states 
reporting few or no WIA self-service-only participants and others 
reporting many hundreds of thousands of such participants.[Footnote 
37] Our analysis of program year 2011 WIASRD data showed that this 
variability persists. Nine states reported that none of their 
participants were self-service-only participants, while 27 states 
reported that over 90 percent of their participants were self-service 
only-participants.[Footnote 38] 

In addition, DOL's guidance requires states to report an occupational 
skills training code for all participants who have completed training, 
but not all state information systems have the capability of capturing 
this code.[Footnote 39] Our analysis of program year 2011 WIASRD data 
showed that states did not report occupational skills training codes 
for about 18 percent of participants in the Adult and Dislocated 
Worker Programs who completed training. As a result of these missing 
data, limited information is available to program managers and DOL 
about the types of occupations for which WIA participants were 
trained. According to officials from DOL's national office, 
occupational skills training codes are underreported because state 
information systems are either programmed with outdated codes or lack 
the capability to report valid codes. 

According to officials from two regional offices, limitations in 
information systems may be the result of states' limited funding and 
technical expertise to upgrade and maintain their information systems. 
For example, state officials we interviewed in five of the eight 
states reported that it is difficult to quickly modify their 
information systems to accommodate new data reporting guidance from 
DOL, which may be due in part to resource constraints. Officials from 
two states and one regional office added that the recent reduction of 
WIA funding for state set-asides, from 15 percent to 5 percent of WIA 
funds, has had a negative effect on states' investment in data 
reporting, including their information systems and technical 
expertise.[Footnote 40] 

As a result of variations in how states collect and report participant 
data for the WIA Adult and Dislocated Worker Programs and limitations 
in their information systems, the actual number of participants in 
these programs is unknown. However, we were able to estimate the 
number of, characteristics of, and services provided to participants 
whose information is recorded by DOL as having been served by the WIA 
Adult and Dislocated Worker Programs using WIASRD data from program 
year 2011. (See additional information on our review of the WIASRD 
data for program year 2011 in appendix II). 

DOL Has Increased Its Oversight Efforts and Taken Steps to Improve 
Data Consistency, but Data Quality Issues Have Not Been Resolved: 

DOL Performed Various Oversight Activities, but It Did Not 
Consistently Use the Results to Improve the Data: 

To improve the quality of data on participants in the WIA Adult or 
Dislocated Worker Programs, DOL has enhanced its oversight efforts and 
introduced new initiatives including having its data contractor 
produce quarterly reports on data issues, requiring states to validate 
their WIA data on an annual basis, and engaging its regional offices 
in periodic reviews of case files from states. However, the agency has 
not established a process to review the results of oversight to 
identify and resolve systemic issues with the quality of participant 
data from the WIA Adult and Dislocated Worker Programs. Our past work 
has shown that the benefit of collecting performance information is 
only fully realized when this information is actually used by managers 
to make decisions oriented toward improving results.[Footnote 41] 

Social Policy Research Associates' Quarterly Reports: 

DOL's contractor, Social Policy Research Associates (SPRA), analyzes 
the WIASRD data that states submit to DOL each quarter and produces 
quarterly reports identifying potential errors in the data reported by 
states and local areas. These reports could also provide useful 
assessments of inaccuracies in the data and potential systemic 
reporting errors.[Footnote 42] However, because not all states are 
aware of or receive copies of these reports from DOL, their value as a 
tool to improve the quality of participant data for the WIA Adult and 
Dislocated Worker Programs is limited. In addition, officials from 
some of the states said that some of the corrections SPRA makes to the 
state WIASRD data files may not be accurate, and may result in 
incorrect data for a state. 

The publicly available data files that SPRA produces are released 
quarterly and include "data issues reports," which identify issues or 
anomalies in the WIA data submitted by each state to DOL. Officials 
from DOL's national office stated that the original intent of SPRA's 
analysis was to create the publicly available data files and not to 
report on data anomalies or errors, even though SPRA has always 
identified issues with the data while creating the publicly available 
files. However, since about 2010, when states started submitting 
quarterly WIASRD files instead of annual ones, reviewing the quality 
of the data and issuing the quarterly "data issues and anomalies 
report" has become standard practice under SPRA's contract, according 
to officials from DOL's national office. The agency's regional offices 
are supposed to provide the states with the published quarterly error 
reports and ask them to update any errors prior to their next 
quarterly WIASRD submission. 

DOL publishes SPRA's data files and reports on its website, and its 
regional offices are expected to share these reports with their states 
so that the states can correct any data issues in their subsequent 
quarterly submissions. However, officials from DOL's Region 5 said 
that although they receive these reports from DOL's national office, 
they have not had the chance to review them or comment on any of the 
errors for states in their region. In addition, officials from DOL's 
Region 1 said that although these reports have always been available 
on DOL's website, they only recently began receiving the reports in a 
user friendly format and that they only recently began providing 
feedback to SPRA on issues identified for the states in their region. 
Our analysis of SPRA's data issues reports suggests that some states 
may not be using SPRA's reports to improve the quality of their data 
on WIA participants, which could be due to a lack of awareness of 
these reports. For example, SPRA's data issue reports for the fourth 
quarters of both program years 2010 and 2011 identified some of the 
same issues, such as errors in the dates of service reported by the 
states.[Footnote 43] This suggests that the states that made these 
errors in 2010 may not have reviewed these reports and used them to 
correct the data reported in the following year. In addition, while 
officials from four states were aware of the SPRA reports, officials 
from four other states told us they either were not aware of these 
reports or that they began receiving these reports from the regions 
only recently, beginning with the final report for program year 2011. 
When asked, officials from DOL's national office stated that they do 
not have a plan to systematically identify or address recurring errors 
noted in SPRA's reports. Government internal control standards note 
that, for oversight and monitoring to be effective, information should 
be recorded and communicated to management and others within the 
entity and external stakeholders, and this should be done within a 
time frame that enables them to carry out their internal control and 
other responsibilities.[Footnote 44] 

Officials added that they do not conduct formal oversight reviews or 
audits of SPRA's data analyses because they consider SPRA to be the 
"data experts" and, therefore, do not know what kind of oversight they 
could provide. According to government standards for internal 
controls, agencies should ensure that ongoing monitoring occurs in the 
course of normal operations, which would include monitoring and 
oversight of contractors through regular management and supervisory 
activities.[Footnote 45] These officials also said they believe any 
major data issues or obstacles would be uncovered by their internal 
data edit checks, which are run on all WIASARD data submitted by the 
states prior to SPRA's review of the data. 

However, according to officials we interviewed in three of the eight 
states we selected, some of the corrections SPRA makes to the state 
WIASRD data files may not be accurate, and may result in incorrect 
data for a state. When SPRA receives the WIASRD data files from DOL, 
it corrects the data after reviewing them for apparent inconsistencies 
or issues stemming from the way states enter the data on individual 
case files. According to SPRA officials, after they receive the data, 
they analyze it for inconsistencies that look odd from a state 
perspective and that would not have been picked up by DOL's internal 
edit checks. For example, SPRA might determine that a missing value 
for a certain variable is equivalent to a "no" response and enter "no" 
or "zero" instead of leaving it blank. However, officials from three 
state workforce agencies told us that they did not always agree with 
the changes SPRA makes during its process of correcting the data. For 
example, officials from California explained that SPRA changed some of 
the numbers entered by local areas for WIASRD variable 325--Employment 
and Training Programs Related to Food Stamps--to zeros because the 
numbers entered seemed too high. According to the state officials, the 
numbers for that variable entered by the local areas were correct. 
States, however, are generally not provided an opportunity to review 
and verify SPRA's changes before they are made, as they only receive 
copies of the data issues reports after they are published. Officials 
from both DOL's national office and from some of the state workforce 
agencies noted that not all issues identified by SPRA represent actual 
errors and that some outliers on certain variables are acceptable. 

States' Annual Data Validation: 

DOL requires each state to validate the data it collects and report on 
participants in the WIA Adult and Dislocated Worker Programs on an 
annual basis, but the findings from these validation efforts have not 
been strategically used to identify systemic issues with or to improve 
the quality of the data on WIA participants.[Footnote 46] Government 
standards for internal controls state that for oversight and 
monitoring to be effective, information should be communicated to 
management and others, along with the use of this information for 
program assessment, so that it can be used to carry out their internal 
control and other responsibilities.[Footnote 47] 

Each year, states are required to review a sample of WIA participant 
records to determine whether the source documents match the 
information in the electronic records states use to collect and report 
data to DOL in WIASRD.[Footnote 48] DOL requires states to validate 
the accuracy of the data they submit annually to ensure that decisions 
about WIA policy and funding are made based on a true picture of the 
number of participants and program outcomes. Although DOL has 
established a provisional 5 percent error rate threshold for states' 
validation of the variables, the agency does not have plans to tie the 
results of these validation efforts to DOL's financial awards or 
penalties because, according to officials from DOL's national office, 
the results of states' validation efforts were never intended to be 
used for enforcement purposes.[Footnote 49] Officials from DOL's 
national office explained that, in their opinion, DOL's regional 
offices should be using the states' validation efforts as a management 
tool to improve the quality of their data by identifying inaccurate or 
confusing variables to target the technical assistance they provide to 
states and local areas. 

DOL, however, does not know what effect state data validation efforts 
have had on the quality of participant data for the WIA Adult and 
Dislocated Worker Programs. In addition, our interviews with regional 
and state officials suggest that DOL's regions and the states are not 
always using the results of these data validation efforts to improve 
data quality or target technical assistance. During our interviews 
with regional and state officials, only those from Region 3 and 
Massachusetts described specific efforts to use the results of the 
state data validation efforts to improve data quality and direct 
technical assistance. Officials from DOL Region 3 noted that the 
region recently began requiring states to respond to findings from 
states' annual data validation efforts and to track error rates found 
in each quarterly submission. Similarly, officials from Massachusetts 
said that they use the results of their data validation efforts to 
direct the technical assistance the state provides to local areas and 
to improve the quality of the state's WIA data. In contrast, officials 
from one state said that, although they receive information about 
errors associated with specific case files as they enter data from 
each file into DOL's data validation system, they do not know how to 
retrieve their state-wide results from DOL's database and they have 
not received any reports from DOL documenting the nationwide results 
of the data validation efforts. Officials from DOL's national office 
said that they were surprised by this, and that data element 
validation results are available to states through DOL's reporting 
system. 

Moreover, our analyses of the results of DOL's efforts to validate 
WIASRD variables for program years 2010 and 2011 suggest that DOL's 
data validation efforts have not prevented high error rates on certain 
data elements for select variables--nationally, error rates for 
certain variables have remained well above the 5 percent threshold 
over both program years. For example, in program year 2010, across all 
states, about 16 percent of the files for the Adult Program for which 
the "date of program exit" variable was reviewed had errors, compared 
to around 14 percent in program year 2011.[Footnote 50] Similarly, the 
nationwide error rate for "date of first staff-assisted core service" 
was above 7 percent in both program years for both the Adult and the 
Dislocated Worker Programs.[Footnote 51] When asked, officials from 
DOL's national office and two regional offices explained that 
variables containing dates frequently have high error rates due to 
discrepancies between the date reported and the date in the source 
documentation. If a date, such as date of dislocation--the date a 
worker lost his or her job--in the hard copy document differs from the 
date in the electronic record, even by one day, the variable for that 
record "fails" the validation check. Moreover, officials from DOL's 
national office and three regional offices stated that high error 
rates resulting from such discrepancies do not necessarily reflect any 
serious issues with the reported data--a participant would still be a 
dislocated worker whether, for example, the date of dislocation was 
June 20th or June 21st. Nonetheless, DOL requires these dates to match 
precisely and errors noted in program year 2011 for these variables 
were still prevalent although similar errors were noted the previous 
year. 

DOL's required annual data validation efforts are resource-intensive 
and time-consuming both for DOL regions and states, according to 
officials from DOL's national office, two DOL regional offices, and 
five states. However, DOL has not yet evaluated the process or 
determined its effect on data quality. Specifically, an official from 
one state workforce agency estimated the cost of its annual data 
validation efforts, including staff time, travel, and other expenses, 
to be about $200,000. Officials in another state explained that they 
would like DOL to reduce the required sample size for the required 
validation of the data elements in WIASRD in order to reduce the 
administrative burden on states. In addition, in 2011 one DOL regional 
office convened a workgroup of representatives from four states that 
analyzed the data validation procedures and provided recommendations 
to DOL's national office for improvements to reduce the administrative 
burden on states. These recommendations included considering using 
alternative sampling methods, revisiting the frequency and precision 
requirements of data validation, and issuing guidance to share "best 
practices" across states and local areas. However, as of September 
2013, DOL had not implemented the workgroup's recommendations, and the 
data validation process remains unchanged. Officials from DOL's 
national office acknowledged the trade-off between monitoring data 
quality and minimizing the administrative burden on states. They said 
that revising WIASRD's edit checks to allow states more flexibility 
may result in making the data validation process more efficient by, 
for example, permitting states to report a range of dates, if 
appropriate, for certain variables. 

Reviews by Regional Offices: 

DOL's regional offices review a sample of case files from states as 
part of their oversight of the quality of data for the WIA Adult and 
Dislocated Worker Programs, but they have not used the results of 
these reviews to identify systemic issues with the quality of the data 
on WIA participants. According to officials from DOL's national 
office, its regional offices are responsible for providing feedback to 
states based on these reviews, and it is not the national office's 
role to conduct any type of systemic review to identify cross-state 
data issues. Government standards for internal controls state that for 
oversight and monitoring to be effective, information should be 
communicated to management and others, along with the use of this 
information for program assessment, so that it can be used to carry 
out their internal control and other responsibilities.[Footnote 52] 

In response to a recommendation from a prior GAO report, DOL began to 
require its regional offices to review a sample of case files to 
monitor states' annual data validation procedures.[Footnote 53] In 
addition, over the past few years, to address concerns about the 
nationwide consistency of monitoring activities, DOL has issued 
additional guidance to its regional offices on the process that should 
be used in reviewing the case files.[Footnote 54] Officials from all 
six of DOL's regional offices reported using these materials when they 
design and conduct their reviews of the case files. The review process 
begins when officials from DOL's regions review a state's most recent 
data validation report and identify a subsample of participant case 
files from the most recent review by the state. In addition to 
checking the data in the electronic records by comparing them to the 
source documentation, DOL's regional staff assess whether the state 
followed the proper procedures in conducting its annual data 
validation efforts, according to officials from DOL's national office. 
The process concludes with a report outlining the DOL regional 
office's findings, including non-compliance with statutory or 
regulatory requirements for collecting and reporting data on WIA 
participants. Many of the reports also identify areas of concern, such 
as when states do not share their annual data validation results with 
local area staff from American Job Center partner programs, and 
identify promising practices observed during the reviews. State 
officials have 45 days to respond to findings in the region's report 
and are also encouraged but are not required to respond to areas of 
concern detailed in the report. In addition, officials from DOL's 
national office noted that states with high error rates on select 
variables are encouraged to inform the regional offices of how they 
plan to reduce their error rates in the future. 

Officials from DOL's national office said that, while they discuss the 
results of these reviews with regional officials and provide state-
specific technical assistance as needed, they do not have a regular, 
formal process for analyzing the findings from these reviews by their 
regional offices, including determining whether similar findings and 
areas of concern were identified across states. DOL officials 
explained that, because the reviews are part of the regional offices' 
oversight of the states, they believe that the national office should 
not be involved in monitoring the results of the reviews or the way in 
which they are conducted. As a result, DOL does not have a systematic 
means of determining the importance of the findings, their prevalence, 
or their likely effect on the quality of the national data on 
participants in the WIA Adult and Dislocated Worker Programs. This 
limits DOL's ability to respond to data issues that are systemic or 
widespread. Table 2 summarizes our analysis of the most prevalent 
issues identified during the most recent reviews for each of the 53 
states and territories. 

Table 2: Most Prevalent Issues Identified During DOL's Regional 
Reviews of States' Case Files, Program Years 2006-2011A: 

Findings or Areas of Concern by Category: Issues relating to 
inadequate, outdated or otherwise limited state information systems; 
Number of Jurisdictions with Finding or Area of Concern (53 total): 26. 

Findings or Areas of Concern by Category: Data reporting; 
Number of Jurisdictions with Finding or Area of Concern (53 total): 47. 

Findings or Areas of Concern by Category: Data reporting; Failure to 
report required information or retain documentation; 
Number of Jurisdictions with Finding or Area of Concern (53 total): 36. 

Findings or Areas of Concern by Category: Data reporting; Issues with 
reported dates of participation, service or exit; 
Number of Jurisdictions with Finding or Area of Concern (53 total): 30. 

Findings or Areas of Concern by Category: Data reporting; Incorrect 
policies or practices regarding participant information, such as 
eligibility; 
Number of Jurisdictions with Finding or Area of Concern (53 total): 20. 

Findings or Areas of Concern by Category: Data reporting; Issues 
relating to sequence of services; 
Number of Jurisdictions with Finding or Area of Concern (53 total): 9. 

Findings or Areas of Concern by Category: Data element validation 
process; 
Number of Jurisdictions with Finding or Area of Concern (53 total): 49. 

Findings or Areas of Concern by Category: Data element validation 
process; Incorrect data validation procedures; 
Number of Jurisdictions with Finding or Area of Concern (53 total): 39. 

Findings or Areas of Concern by Category: Data element validation 
process; High error rates on one or more data elements; 
Number of Jurisdictions with Finding or Area of Concern (53 total): 37. 

Findings or Areas of Concern by Category: Data element validation 
process; Insufficient documentation; 
Number of Jurisdictions with Finding or Area of Concern (53 total): 23. 

Findings or Areas of Concern by Category: Data element validation 
process; Failure to use results to make improvements in data quality; 
Number of Jurisdictions with Finding or Area of Concern (53 total): 9. 

Source: GAO analysis of DOL's case file review reports, program years 
2006-2011. 

[A] While the most recent review for Arkansas looked at data from 
program year 2006, most of the reviews looked at data from program 
years 2008, 2009, 2010, or 2011. In addition, most reviews covered 
either 1 or 2 years of data. 

[End of table] 

Technical Assistance: 

Over the past few years, DOL has issued additional guidance and 
provided technical assistance to states and local areas, including 
training and webinars, to clarify and explain the requirements for 
collecting and reporting data for the WIA Adult and Dislocated Worker 
Programs. However, some state officials said that DOL's technical 
assistance is not always timely, and that DOL could do more to 
facilitate the collection or sharing of promising data collection and 
reporting practices across regions and states. 

For example, DOL has provided general technical assistance on data 
reporting for the WIA Adult and Dislocated Worker Programs to states, 
and officials from three of the eight states said that the assistance 
provided by DOL's regional offices was useful in helping them address 
some of the challenges related to data reporting. In particular, 
officials from DOL's national office said that some regional offices 
issue quarterly performance letters to states that include program 
year performance data and any related analysis, in addition to hosting 
quarterly phone conferences with state performance specialists to 
discuss performance issues. DOL also sponsors the Workforce3one 
website, which contains a variety of training and background materials 
related to data collection and reporting for WIASRD. National and 
regional DOL data specialists also said they hold biweekly meetings to 
discuss data issues. In addition, DOL officials from each region 
described regular communication they have with state officials to 
provide technical assistance in response to specific data reporting 
issues, and officials from five of the six regional offices described 
conference calls that they have hosted as opportunities for states to 
discuss challenges related to the quality of their data on WIA 
participants. Furthermore, DOL's national and regional offices have 
access to an internal data system, Infospace, which allows them to 
retrieve and review publicly available WIASRD data by state and local 
area. 

Over the past few years, DOL has also issued a number of Training and 
Employment Guidance Letters and Training and Employment Notices 
related to data collection and reporting for WIA. In 2011, DOL 
developed and hosted a series of webinars for states and local areas, 
including one on data validation and data quality issues.[Footnote 55] 
The materials from this session describe the approach DOL uses to 
monitor the data and explore issues and findings across states. They 
also highlight how data validation can be used to improve data systems 
and provide information on the guidance regional offices use to review 
states' case files. While the webinar materials present information on 
some of the consistent findings across states--such as incorrect 
source documentation and exit dates--DOL officials said that they have 
not provided any additional formal technical assistance to address 
these issues on a national level because of resource limitations. 
Moreover, they said it would be difficult for DOL to provide such 
assistance without having first reviewed the results of its own 
monitoring efforts. DOL's webinar also noted that DOL would establish 
a working group in the summer of 2011 to look into and possibly revise 
the source documentation requirements for the annual data validation 
process. However, as of the summer of 2013, DOL officials said that 
this working group had not been established due to competing 
priorities and resource constraints. 

Officials from two states noted that DOL's updates to its guidance for 
collecting and reporting the data are often not provided far enough in 
advance to be implemented by the time changes take effect. For 
example, officials from California said that they often do not have 
enough lead time to properly implement new data elements or guidance 
when it is issued by DOL. According to state officials, the state 
generally has 1 month or less to change its automated system to meet 
deadlines, which is not enough time. In addition, officials from 
Georgia said the data validation reports they currently receive from 
DOL are not timely since they are at least 3 months old by the time 
the state receives the reports. Some state officials also told us that 
they would benefit from learning how their peers are addressing 
challenges in reporting data on the WIA programs. Officials from DOL's 
national office, however, told us they do not currently facilitate the 
collection or sharing of promising data collection and reporting 
practices across regions or states, and have no plans to do so--
because of competing resources, the agency's main focus is on service 
delivery rather than data collection. Best practices state that high-
performing organizations continually assess performance and efforts to 
improve performance. In particular, managers can use performance 
information to identify and share more effective processes and 
approaches.[Footnote 56] 

DOL Has Taken Steps to Improve Data Consistency, but It Has Not 
Evaluated These Efforts: 

In addition to its regular monitoring activities, DOL has taken 
specific steps to improve the consistency of the WIA data collected by 
states and local areas. For example, the agency has developed an 
integrated data reporting system, which is being piloted in two 
states.[Footnote 57] However, DOL has not evaluated the results of the 
pilot program to determine whether it has had a positive effect on the 
quality of participant data for the WIA Adult and Dislocated Worker 
Programs and, despite not having evaluated its effectiveness, plans to 
expand the program to additional states. 

To standardize and streamline reporting across several of DOL's 
workforce programs--the Wagner-Peyser Program, the WIA Adult, 
Dislocated Worker and Youth Programs, Veterans Employment and Training 
Service, National Emergency Grants, and Trade Adjustment Assistance 
Programs--the agency has developed an integrated data reporting 
system, WISPR. Two states, Pennsylvania and Texas, have been piloting 
WISPR since 2007 to collect and report data on the WIA Adult and 
Dislocated Worker Programs, and a third state--Utah--plans to start 
piloting WISPR in the fall of 2013. Officials from DOL's national 
office and from Utah explained that one of WISPR's key advantages over 
the current separate reporting systems for each program is that WISPR 
has standardized variables that include all the required variables for 
each program.[Footnote 58] Utah officials stated that they believe 
WISPR will be an improvement over their current system because future 
changes in DOL's guidance for any of the workforce programs it 
administers--including the WIA Adult and Dislocated Worker Programs--
would be incorporated into a single system, facilitating 
implementation of these changes. 

To facilitate a future transition to WISPR in the remaining states, 
DOL has obtained approval from the Office of Management and Budget to 
modify the current record layout of WIASRD to match that of WISPR. 
[Footnote 59] DOL officials told us they expect to implement the 
revised WIASRD record layout in the fall of 2013. However, as of 
August 2013, they said it is not clear whether or when nationwide 
implementation of WISPR will occur because this depends on the 
resources available to upgrade both federal and state information 
systems and the associated programming costs. While WISPR appears to 
offer advantages over the current reporting system that might make it 
a promising step forward, DOL does not currently have plans to 
evaluate the results of the pilot program to determine whether it has 
had a positive effect on the completeness and consistency of 
participant data for the WIA Adult and Dislocated Worker Programs 
before expanding the program to other states. DOL officials cited the 
agency's limited resources as the reason for not planning an 
evaluation of the WISPR pilot program before expanding it to other 
states. However, best practices note that evaluation can play a key 
role in program planning, management, and oversight by providing 
feedback to program managers, legislative and executive branch policy 
officials, and the public. Further, when pilot programs are designed 
to produce change--such as by allowing for more streamlined data 
collection and reporting--assessing the impact is essential for 
knowing if the pilot is meeting its goals.[Footnote 60] Without an 
evaluation of WISPR, DOL will not know if this data system has 
resulted in the collection of more accurate WIA participant data when 
compared to WIASRD.[Footnote 61] 

Finally, DOL administers two grant programs that states can use to 
improve their WIA information systems: the Workforce Innovation Fund 
and the Workforce Data Quality Initiative.[Footnote 62] DOL's 
Workforce Innovation Fund is a competitive grant program that supports 
innovative approaches to the design and delivery of employment and 
training services. Although it is not targeted specifically at 
information systems, at least 3 of the 26 grants awarded by DOL have 
been used for local initiatives to integrate their workforce data 
systems. For example, one local area we visited in Chicago received a 
Workforce Innovation Fund grant that they plan to use to integrate the 
different data systems used by the workforce programs in their local 
area. Officials said that they hope the improvements will result in 
more data-driven decisions about service delivery, and that all 
employment and training programs in their local area will be able to 
share information electronically. Another grant program, DOL's 
Workforce Data Quality Initiative, is also not specifically aimed at 
WIA data reporting but may have positive incidental effects on WIA 
data quality, according to DOL officials. The purpose of this 
initiative is to create a longitudinal database to chart individuals' 
progress through the education system and beyond to the labor market. 
This effort would entail upgrades to state information systems that 
may resolve some data reporting issues currently attributed to limited 
technological capacities. At this time, it is too early to know 
whether these grants will have a positive effect on the quality of WIA 
participant data. 

Conclusions: 

Collecting and reporting consistent and complete data is important for 
program oversight and management and to evaluate the effectiveness of 
program activities and services, but it can be difficult when federal 
programs are carried out in partnership with states and local areas. 
DOL has taken steps to improve the quality of the data on WIA's Adult 
and Dislocated Worker Programs. However, the flexibilities in DOL's 
guidance, reflective of those inherent in the programs' authorizing 
statute, which allows states flexibility in program design, along with 
limitations in state information systems, present challenges to DOL in 
collecting and reporting consistent and complete data on a unique 
count nationwide of participants in the WIA Adult and Dislocated 
Worker Programs. Without such data, policymakers, program officials, 
and other stakeholders have an incomplete picture of the number of 
adults and dislocated workers served, their characteristics, and the 
type and level of services received. 

In addition, while DOL engages in several types of oversight 
activities designed to ensure the accuracy of states' data on 
participants in the WIA Adult and Dislocated Worker Programs, it does 
not consistently share the results of its oversight activities with 
states and local areas. As a result, states and local areas are not 
always aware of potential data quality issues, and may miss 
opportunities to improve their data collection and reporting. 
Moreover, since 2007, two states have been piloting a new information 
system that tracks program participants across several of DOL's 
employment and training programs, but DOL does not plan to evaluate 
its effects on the quality of the data collected on participants in 
the WIA Adult or Dislocated Worker or other programs before it expands 
the system to other states. Similarly, DOL does not regularly collect 
and disseminate promising practices to states and local areas, which 
could facilitate the adoption of steps other states and local areas 
have taken to improve their data collection and reporting efforts. 
While it may not be possible to achieve 100 percent precision and 
accuracy in the data reported on participants in a large, complex 
system like the workforce investment system, by not appropriately 
targeting their available resources and facilitating sharing of 
promising practices among states to continuously try to improve the 
quality of the data, DOL misses an opportunity to identify and address 
longstanding, systemic issues. 

Recommendations for Executive Action: 

1. To improve the consistency and completeness of national data on 
participants in the WIA Adult and Dislocated Worker Programs, we 
recommend that the Secretary of Labor take additional steps to improve 
the uniformity of participant data reported by states. These steps 
could include the following: 

a. providing additional guidance to states on data reporting, such as 
how core and intensive services should be recorded for WIA 
participants who receive these services through partner programs; and: 

b. conducting an evaluation or review of WISPR to determine if it has 
resulted in more complete and consistent data collection and reporting 
for participants in the WIA Adult and Dislocated Worker Programs and 
placing a high priority on the implementation of WISPR if it is shown 
to improve data consistency and completeness. 

2. We also recommend that the Secretary of Labor promote a formal, 
continuous process for improving the quality of data on participants 
in the WIA Adult and Dislocated Worker Programs through such measures 
as the following: 

a. consistently sharing the results of all oversight activities with 
states and local areas, including findings from validation of 
participant data; 

b. reviewing the methods used for data validation, such as its scope 
and error rate threshold, to identify opportunities to increase 
efficiencies and accountability in the process. This could include 
implementing, if appropriate, recommendations from the Regions' review 
of data validation procedures; 

c. evaluating data validation efforts to determine their effects on 
data quality, particularly on systemic errors, and providing targeted 
guidance and assistance to states and local areas to address such 
errors; 

d. regularly monitoring Social Policy Research Associates' corrections 
and analyses of state WIA participant data, sharing this information 
with states, and coordinating with states to ensure that any 
corrections are appropriate and accurate; and: 

e. collecting and disseminating promising practices to states and 
local areas on data collection and reporting on a regular basis. 

Agency Comments and Our Evaluation: 

We provided a draft of this report to officials at DOL for their 
review and comment. We received written comments from DOL, which are 
reproduced in their entirety in appendix III. DOL officials did not 
state whether they agreed or disagreed with our recommendations. These 
officials acknowledged the importance of having reliable data to 
effectively manage and evaluate the WIA Adult and Dislocated Worker 
Programs; however, they commented that data reliability should be 
balanced with the flexibility WIA gives to states and with DOL's 
responsibility to prioritize use of its limited resources. They stated 
that WIA provides states and local areas with the flexibility to serve 
their customers in the way that best suits their particular needs. DOL 
officials also stated that the agency has invested significant 
resources in its workforce performance accountability system, 
especially for WIA programs. According to officials, the agency has a 
robust system in place to ensure data quality and reliability and has 
recently made several enhancements to the reporting system. In their 
comments, DOL officials detailed various efforts they plan to take to 
address our recommendations. Nonetheless, we believe that these 
efforts will not sufficiently address the specific data quality issues 
we identified and encourage DOL to take more targeted steps as 
outlined in our recommendations. 

In response to our first recommendation, DOL officials said that they 
believe the agency's current guidance is clear but that they will 
continue to work with states to develop additional guidance, as 
necessary, such as forthcoming guidance on how to avoid duplication of 
services when co-enrolling participants across multiple partner 
programs. However, it is important that any additional guidance also 
specify when to count job seekers as WIA participants if they also 
receive services funded by partner programs. We also encourage DOL to 
develop additional guidance for the WIASRD variables noted in our 
report that are open to interpretation, such as "type of training," to 
facilitate consistent reporting on participants in these programs. DOL 
officials also noted that an evaluation of WISPR is subject to the 
agency's resource constraints, adding that the purpose of WISPR was 
never explicitly to improve data quality. As we stated in our report, 
however, WISPR seeks, in part, to improve the consistency of WIA data 
by standardizing reporting across the workforce system. As such, it 
has the potential to improve data quality. Therefore, we encourage DOL 
to evaluate the system in order to make an informed decision on how 
best to allocate finite agency resources going forward. 

In response to our second recommendation, DOL officials stated that 
they consistently share the results of the agency's oversight 
activities with states but acknowledged that more could be done to 
analyze the results of its activities to identify and share similar 
findings and areas of concern across multiple states. DOL officials 
added that they will work with the regional offices towards this goal. 
With regard to the agency's data validation methods, officials said 
they regularly review these methods and solicit input from states on 
how to improve them. Specifically, DOL pointed out that, as required 
by the Paperwork Reduction Act, the Office of Management and Budget 
reviews DOL's data validation process every 3 years and solicits 
public comment before approving the methodology and authorizing data 
collection, and that their 2014 submission will reflect state input. 
However, given the time-consuming nature of data validation, we 
believe the agency should take additional actions to review its 
current methods specifically with an eye toward making them more 
efficient and holding states accountable for their data validation 
results. 

With regard to evaluating the effectiveness of its data validation 
efforts, DOL officials said that the agency plans to consider the 
regional data validation workgroup's findings and recommendations from 
2011, explore ways to streamline the process, and examine the effect 
of data validation on error rates. We commend DOL's plans, but to 
adequately address the persistently high data error rates we found in 
our analysis, we believe it is necessary to go beyond evaluating the 
effectiveness of its data validation efforts and pinpoint the 
underlying cause of the errors so that they can be addressed. 

Officials also pointed out that the agency already monitors and shares 
the analyses of state data conducted by its contractor, Social Policy 
Research Associates (SPRA). They stated that SPRA's corrections of 
states' data have been publicly available with the data set from the 
inception of WIA. They also noted that since program year 2011 DOL has 
provided SPRA's analyses and corrections to the states (through ETA 
Regional Offices) on a quarterly basis for states to either correct or 
dispute. They noted that this is a formal and recurring process, and 
that ETA Regional Offices have begun to analyze state date WIASRD data 
on a regular basis as part of their annual review cycles. However, we 
found that not all states are aware of or receive copies of SPRA's 
reports, and that some of the corrections SPRA makes to the state 
WIASRD data files may not be accurate. Furthermore, as noted in our 
report, DOL officials told us that they do not conduct formal 
oversight reviews or audits of SPRA's data analyses. 

In addition, DOL officials reiterated that data collection and 
reporting are topics that are included in workforce3one, a point we 
noted in our report. However, we maintain that DOL could do more to 
facilitate the sharing of information across states, such as creating 
a forum through which states could learn how their peers are 
addressing challenges in data reporting for participants in the WIA 
programs. Finally, DOL provided technical comments, which we 
incorporated, as appropriate. 

As agreed with your offices, unless you publicly announce the contents 
of this report earlier, we plan no further distribution until 30 days 
from the report date. At that time, we will send copies to the 
appropriate congressional committees, the Secretary of Labor, and 
other interested parties. In addition, the report will be available at 
no charge on the GAO website at [hyperlink, http://www.gao.gov]. 

If you or your staff have any questions concerning this report, please 
contact me at (202) 512-7215 or moranr@gao.gov. Contact points for our 
Offices of Congressional Relations and Public Affairs may be found on 
the last page of this report. Key contributors to this report are 
listed in appendix IV. 

Signed by: 

Revae Moran, Director: 
Education, Workforce, and Income Security Issues: 

[End of section] 

Appendix I: Scope and Methodology: 

Our objectives were to determine: (1) what factors have affected the 
ability to report consistent and complete data on participants in the 
Workforce Investment Act (WIA) Adult and Dislocated Worker Programs, 
and (2) what actions has the Department of Labor (DOL) taken to 
improve the quality of participant data. To address our objectives, we 
reviewed applicable laws and regulations, as well as DOL's guidance to 
states for collecting and reporting data on participants in the WIA 
Adult and Dislocated Worker Programs.[Footnote 63] We also interviewed 
officials from DOL's Employment and Training Administration, its 
Office of Inspector General, and its six regional offices. In 
addition, we visited or telephoned a nongeneralizable sample of eight 
states. Within each state, we visited or contacted at least one 
American Job Center--formerly known as a one-stop center--or a local 
workforce board. We also assessed the reliability of program year 2011 
data from the Workforce Investment Act Standardized Record Data 
(WIASRD) system by testing the data electronically and interviewing 
knowledgeable agency officials and DOL's data contractor.[Footnote 64] 
We found the data in appendix II to be sufficiently reliable for the 
purposes of providing estimates of the number of, characteristics of, 
and services provided to participants whose information is recorded by 
DOL as having received services from either the WIA Adult Program or 
the WIA Dislocated Worker Program. The data are not reliable for other 
purposes, such as making state-to-state comparisons, because of 
variations in how states collect and report data on participants in 
the WIA Adult and Dislocated Worker Programs. 

Selected States: 

To better understand the data reporting challenges faced by states and 
the actions DOL has taken to improve the quality of participant data, 
we interviewed officials from a nongeneralizable sample of eight 
states.[Footnote 65] Within each state, we visited or contacted at 
least one American Job Center or the local workforce entity. We 
conducted in-person interviews with DOL officials in Regions 1 
(Boston), 3 (Atlanta), and 5 (Chicago); state and local workforce 
officials in California, Georgia, Illinois, Massachusetts, and 
Washington; and American Job Center officials in Maryland. We 
conducted telephone interviews with DOL officials in Regions 2 
(Philadelphia), 4 (Dallas), and 6 (Sacramento), and with state 
workforce officials in Maryland, South Dakota, and Utah.[Footnote 66] 
We nonstatistically selected these states to provide diversity on the 
basis of: (1) geographic location, (2) total federal spending on the 
Adult and Dislocated Worker Programs in program year 2010, (3) the 
extent of data issues identified in the fourth quarter of program year 
2010, (4) whether the state reported participants who only received 
core self-services, and (5) the number of local areas within the 
state.[Footnote 67] 

Table 3: Selected Site Visit States and Local Areas: 

State: California; 
Local Area: Sacramento City/County Consortium; 
City: Sacramento. 

State: California; 
Local Area: San Joaquin County; 
City: Stockton. 

State: Georgia; 
Local Area: Cobb County; 
City: Marietta. 

State: Georgia; 
Local Area: Fulton County; 
City: Sandy Springs. 

State: Illinois; 
Local Area: Cook County; 
City: Chicago. 

State: Maryland; 
Local Area: Prince George's County; 
City: Largo. 

State: Massachusetts; 
Local Area: Western Massachusetts; 
City: Springfield. 

State: South Dakota. 

State: Utah. 

State: Washington; 
Local Area: King County; 
City: Seattle. 

Source: GAO state and local area interviews. 

[End of table] 

In each state, we obtained general information about the state's and 
the local area's implementation of the WIA Adult and Dislocated Worker 
Programs and on any challenges they may have encountered in collecting 
and reporting data on program participants. We also asked about 
actions DOL has taken to improve the quality of the data on 
participants in the WIA Adult and Dislocated Worker Programs. We used 
semi-structured interviews for our regional, state, and local 
interviews. Because we interviewed officials from a nongeneralizable 
sample of eight states and selected local areas, we cannot generalize 
our findings beyond the data collected on those states and local areas. 

Analysis of DOL's Participant Data: 

To assess the reliability of DOL's data in the WIASRD database for 
participants in the WIA Adult and Dislocated Worker Programs, we (1) 
reviewed existing documentation related to the data sources, including 
reports issued by DOL's Office of Inspector General; (2) 
electronically tested the WIASRD data to identify potential problems 
with consistency, completeness, or accuracy; and (3) interviewed DOL's 
data contractor and knowledgeable agency officials to obtain 
information about the data. 

Our electronic testing consisted of identifying inconsistencies, 
outliers, missing values, and other errors. More specifically, the 
electronic testing included assessing the reliability of data 
collected on the characteristics and the services participants in the 
WIA Adult and Dislocated Worker Programs received in program year 
2011.[Footnote 68] Prior to testing the data, we combined 160 records 
that were overlapping or duplicative into 80 unique records and 
removed 19 records that had missing or erroneous participation dates. 
A few variables, including data on a participant's dislocation date--
the date a worker lost his or her job, and the occupational codes for 
participants who completed training, were found to not be sufficiently 
reliable for our purposes and were not included in our report. In 
addition, we analyzed the publicly available WIASRD data file for 
program year 2011, which was produced for DOL by its data contractor, 
Social Policy Research Associates. As part of our analysis, we 
reviewed the steps the data contractor took to correct the data and, 
to the extent possible, compared our data to the publicly available 
file. In light of variations in how states collect and report 
participant data for the WIA Adult and Dislocated Worker Programs and 
limitations in their information systems, the actual number of 
participants in these programs is unknown. However, we were able to 
estimate the number of, characteristics of, and services provided to 
participants whose information is recorded by DOL as having been 
served by the WIA Adult and Dislocated Worker Programs using WIASRD 
data from program year 2011. 

Analysis of DOL's Oversight and Monitoring Efforts: 

To describe and assess DOL's oversight and monitoring efforts, we 
reviewed technical assistance guides and material posted to 
Workforce3One, including DOL's Core Monitoring guides and Data 
Validation Reporting System guidance. We also interviewed officials 
from DOL's Employment and Training Administration national office and 
from all of DOL's six regional offices. In addition, we obtained and 
reviewed copies of DOL's monitoring reports, including the results of 
DOL's program year 2010 and 2011 annual data validation efforts and 
the most recent case file review for each state and territory. 
[Footnote 69] To analyze the results of the annual data validation 
efforts, we calculated the average reported error rate for each 
variable across states.[Footnote 70] We included in our analysis all 
variables on characteristics and services.[Footnote 71] To analyze 
trends in the results of the case file reviews, we reviewed the 
findings and areas of concern identified in each review and 
categorized them to identify common issues present in multiple states. 

We conducted this performance audit from August 2012 through November 
2013 in accordance with generally accepted government auditing 
standards. Those standards require that we plan and perform the audit 
work to obtain sufficient, appropriate evidence to provide a 
reasonable basis for our findings and conclusions based on our audit 
objectives. We believe that the evidence obtained provides a 
reasonable basis for our findings and conclusions based on our audit 
objectives. 

[End of section] 

Appendix II: Data on Participants in the WIA Adult and Dislocated 
Worker Programs for Program Year 2011: 

We reviewed the data collected by the Department of Labor (DOL) in the 
Workforce Investment Act Standardized Record Data (WIASRD) system on 
the number of, characteristics of, and services provided to 
participants in the WIA Adult and Dislocated Worker Programs in 
program year 2011.[Footnote 72] We found the data to be sufficiently 
reliable for the purposes of providing estimates of the number of, 
characteristics of, and services provided to, participants whose 
information is recorded by DOL in WIASRD. These estimates are 
presented in figures 4 through 9. 

Figure 4: Total Participants Enrolled in the Adult and Dislocated 
Worker Programs, Program Year 2011: 

[Refer to PDF for image: overlapping circles] 

Adult Program: 6,919,910; 
Dislocated Worker Program: 1,157,466; 
Enrolled in both programs: 530,703. 

Source: GAO analysis of Department of Labor participant data. 

[End of figure] 

Figure 5: Highest Level of Service Received by Adult and Dislocated 
Worker Program Participants, Program Year 2011[A]: 

[Refer to PDF for image: stacked vertical bar graph] 

Adult Program: 
Core self-services and informational activities: 75%; 
Core staff-assisted services: 16%; 
Intensive services: 5%; 
Training: 3%. 

Dislocated Worker Program: 
Core self-services and informational activities: 5%; 
Core staff-assisted services: 51%; 
Intensive services: 24%; 
Training: 20%. 

Source: GAO analysis of Department of Labor participant data. 

[A] Numbers may not add to 100 percent due to rounding. Nine states 
reported that none of their Adult Program participants received only 
core self-services and informational activities and two states 
reported that no participants in their Adult or Dislocated Worker 
Program received staff assisted core services. 

[End of figure] 

Figure 6: Duration of Training for Adult and Dislocated Worker Program 
Participants Who Completed or Withdrew from Training, Program Year 
2011[A]: 

[Refer to PDF for image: stacked vertical bar graph] 

Participants in Adult Program: 
Time spent in training: 
Less than 4 weeks: 17,900; 12%; 
4 to 13 weeks: 39,800; 26%; 
14 to 26 weeks: 28,200; 18%; 
27 to 52 weeks: 29,200; 19%; 
53 to 104 weeks: 28,300; 18%; 
More than 104 weeks: 11,500; 7%. 

Participants in Dislocated Worker Program: 
Time spent in training: 
Less than 4 weeks: 12,800; 8%; 
4 to 13 weeks: 32,000; 20%; 
14 to 26 weeks: 27,000; 17%; 
27 to 52 weeks: 31,000; 20%; 
53 to 104 weeks: 39,200; 25%; 
More than 104 weeks: 15,900; 10%. 

Source: GAO analysis of Department of Labor participant data. 

[A] Our analysis was restricted to participants who completed or 
withdrew from training. 

[End of figure] 

Figure 7: Type of Training Services Provided to Adult and Dislocated 
Worker Program Participants, Program Year 2011[A]: 

[Refer to PDF for image: stacked vertical bar graph] 

Participants in Adult Program: 
Type of training services: 
On-the-job training: 8%; 
Skill upgrading and retraining: 14%; 
Entrepreneurial training: less than 1%; 
ABE or ESL in combination with training: 3%; 
Customized training: 4%; 
Other occupational skills training: 71%. 

Participants in Dislocated Worker Program: 
Type of training services: 
On-the-job training: 8%; 
Skill upgrading and retraining: 15%; 
Entrepreneurial training: less than 1%; 
ABE or ESL in combination with training: 2%; 
Customized training: 1%; 
Other occupational skills training: 74%. 

Source: GAO analysis of Department of Labor participant data. 

[A] Our analysis was restricted to participants who received training. 

[End of figure] 

Figure 8: Percentage of Adult and Dislocated Worker Program 
Participants Registered in Selected Partner Programs, Program Year 
2011[A]: 

[Refer to PDF for image: comparison graph] 

Adult Program: 
Percentage of total participants: 
Wagner-Peyser: 83%; 
Other partner programs: 17%; 
Only WIA: 17%; 
Trade Adjustment Assistance: less than 1%. 

Dislocated Worker Program: 
Percentage of total participants: 
Wagner-Peyser: 84%; 
Other partner programs: 24%; 
Only WIA: 12%; 
Trade Adjustment Assistance: 5%. 

Source: GAO analysis of Department of Labor participant data. 

[A] Other partner programs include the Adult Education, Job Corps, 
National Farmworker Jobs, Indian and Native American, Veterans, 
Vocational Education, Vocational Rehabilitation, Youthbuild, Title V 
Older Worker, and Employment and Training Services Related to Food 
Stamps Programs. States are not required to report whether 
participants in the WIA Adult or Dislocated Worker Program received 
services from these partner program. Seventeen states reported that 
virtually no WIA Adult Program participants received services from any 
of these programs. 

[End of figure] 

Figure 9: Adult and Dislocated Worker Program Participant 
Demographics, Program Year 2011A: 

[Refer to PDF for image: stacked vertical bar graph] 

Gender: 
Adult Program: 
Men: 51%; 
Women: 49%; 
Dislocated Worker Program: 
Men: 51%; 
Women: 49%. 

Education: 
Adult Program: 
Less than high school diploma: 12%; 
High school diploma or equivalent: 45%; 
Some post high school: 20%; 
College degree or more: 23%. 
Dislocated Worker Program: 
Less than high school diploma: 8%; 
High school diploma or equivalent: 42%; 
Some post high school: 22%; 
College degree or more: 28%. 

Ethnicity and race: 
Adult Program: 
Hispanic: 11%; 
American Indian or Alaskan Native: 3%; 
Asian: 2%; 
Native Hawaiian or Pacific Islander: 1%; 
White: 63%; 
Did not self-identify: 9%; 
Dislocated Worker Program: 
Hispanic: 12%; 
American Indian or Alaskan Native: 2%; 
Asian: 3%; 
Native Hawaiian or Pacific Islander: 1%; 
White: 70%; 
Did not self-identify: 4%. 

Source: GAO analysis of Department of Labor participant data. 

[A] Participants in the WIA Adult and Dislocated Worker Programs who 
did not disclose their gender were not included in the percentages 
listed above for gender. About 11 percent of participants in the Adult 
Program and less than 1 percent of participants in the Dislocated 
Worker Program did not report their gender. Percentages of ethnicity 
and race do not add up to 100 because individuals can identify in more 
than one category. 

[End of figure] 

[End of section] 

Appendix III: Comments from the Department of Labor: 

U.S. Department of Labor: 
Assistant Secretary for Employment and Training: 
Washington, D.C. 20210: 

November 1, 2013: 

Revae E. Moran: 
Director: 
Education, Workforce, and Income Security Issues: 
U.S. Government Accountability Office: 
441 G Street, NW: 
Washington, District of Columbia 20548: 

Dear Ms. Moran: 

On behalf of the U.S. Department of Labor (DOL), I want to thank you 
for the opportunity to review the Government Accountability Office's 
(GAO) draft report entitled: Workforce Investment Act: DOL Should Do 
More to Improve the Quality of Its Participant Data (GAO-14-4). We 
appreciate the information provided in the report and the opportunity 
to comment. 

We agree that ensuring the reliability of data reported by the states 
is important to effectively manage and evaluate Workforce Investment 
Act (WIA) Adult and Dislocated Worker programs; however, we believe it 
should be balanced with the flexibility that WIA gives to states and 
with the Department's responsibility to prioritize use of its limited 
resources. WIA provides states and local areas with the flexibility to 
serve their customers in the way that best suits their particular 
needs. Under WIA, states also have flexibility to design their 
management and information systems. As such, there is no national 
management and information system that states are required to adopt. 
As a result, states vary in how they design their programs, how they
fund their services, and the level of integration across programs 
within their management information systems. 

The Department has invested significant resources in its workforce 
performance accountability system, especially for WIA programs. The 
Department has a robust system in place to ensure data quality and 
reliability, and has recently made several enhancements to the 
reporting system. Beginning with Program Year 2013, all states will 
upload their WIA participant data files into the Enterprise Data 
Reporting and Validation System (EDRVS), where each file will go through
over 200 edit checks before being accepted. The system also will 
calculate performance measures and create state reports to ensure 
uniformity in reporting. States also conduct a Data Element Validation 
review where participant records are validated against source 
documentation to ensure data integrity. 

The GAO report makes two recommendations to the Department. The 
Department's responses to these recommendations are outlined in more 
detail below. 

GAO Recommendation 1: 

To improve the consistency and completeness of national data on 
participants in the WIA Adult and Dislocated Worker Programs, GAO 
recommends that the Secretary of Labor take additional steps to 
improve the uniformity of participant data reported by states. 

GAO Recommendation 1a: 
Providing additional guidance to states on data reporting, such as how 
core and intensive services should be recorded/or WIA participants who 
receive these services through partner programs. 

The Department believes its guidance is clear on reporting these 
services, based on the definitions of core and intensive at WIA 134( 
d)(2) and (3) and at 20 CFR 663.150. Further, additional guidance 
regarding how states report services rendered by partner programs is 
covered in the Employment and Training Administration's Training and
Employment Guidance Letter (TEGL) 17-05, "Common Measures Policy for the
Employment and Training Administration's (ETA) Performance 
Accountability System and Related Performance Issues." In that TEGL, 
states are encouraged to report services provided by partner programs, 
although it is not mandated because there is no statutory authority to 
report such information. Also, it is important to note that the WIA 
reporting system is focused on services funded by WIA; the 
Department's responsibility is to report on WIA-funded services. The 
Department will continue to work with states on this issue and develop 
additional guidance as necessary. Further guidance is forthcoming on 
how to avoid duplicating services when co-enrolling participants across
multiple programs. 

GAO Recommendation 1b:
Conducting an evaluation or review of WISPR to determine if it has 
resulted in more complete and consistent data collection and 
reporting/or participants in the WIA Adult and Dislocated Worker 
Programs and placing a high priority on the implementation of 
WISPR if it is shown to improve data consistency and completeness. 

The Department shares GAO's interest in the Workforce Investment 
Streamlined Performance Reporting System (WISPR). Two states have been 
using WISPR to report to DOL since 2007, and additional states 
continue to express interest in implementing use of the WISPR. WISPR 
was designed to facilitate integrated reporting across the various
workforce programs. The Department notes, however, the purpose of 
WISPR was never explicitly to improve data quality, but to ease the 
reporting burden placed on states. ETA is in the process of modifying 
the record layouts of multiple program data sets to align with WISPR 
to encourage its future expansion. For example, ETA has already
modified WIA and Trade Adjustment Assistance record layouts to align 
with WISPR and plans to do the same with Wagner-Peyser. As noted in 
the report, evaluation and expansion of WISPR are subject to resource 
constraints. 

GAO Recommendation 2: 

GAO also recommends that the Secretary of Labor promote a formal, 
continuous process for improving the quality of data on participants 
in the WIA Adult and Dislocated Worker Programs through such measures 
as the following: 

GAO Recommendation 2a: 
Consistently sharing the results of all oversight activities with 
states and local areas, including findings from validation of 
participant data. 

We agree with this recommendation. The Department consistently shares 
with each individual state its own data validation results that are 
initially submitted by each state. However, we acknowledge that more 
can be done to analyze data validation results to identify and share 
similar findings and areas of concern across multiple states, and we
will work with our Regional Offices towards this goal. 

GAO Recommendation 2b: 
Reviewing the methods used for data validation, such as its scope and 
error rate threshold, to identify opportunities to increase 
efficiencies and accountability in the process. This could include 
implementing, if appropriate, recommendations from the Regions' review 
of data validation procedures. 

The Department shares GAO's interest in reviewing the methods used for 
data validation and already participates regularly in such a review. 
The Department solicits input from states on changes to the 
methodology for conducting data validation and to identify 
opportunities to increase efficiencies and accountability in the 
process. The Office of Management and Budget (OMB) reviews the 
Department's data validation process every three years, under the 
requirements of the Paperwork Reduction Act. As part of this process, 
OMB solicits public comment on the data validation approach. OMB's
decisions to approve the methodology and authorize the data collection 
are based on its view of the balance between the costs (burden) and 
benefits (data quality). The Department will submit an Information 
Collection Request to have the data validation process reauthorized in 
the fall of 2014 that reflects state input. 

GAO Recommendation 2c: 
Evaluating [previously conducted] data validation efforts to determine 
their effects on data quality, particularly on systemic errors, and 
providing targeted guidance and assistance to states and local areas 
to address such errors. 

The Department appreciates GAO's interest in evaluating the impacts 
that data validation has had on data quality and error rates. Data 
validation was implemented in 2003 and has equipped the Department 
with the information necessary for assuring stakeholders that
statutorily required state reports are computed consistently and 
accurately across the states. Data validation also has provided the 
Department with timely information on the quality of the underlying 
information on individual participants across the states. The
Department will consider the regional data validation workgroup's 
findings and recommendations from 2011, explore other ways to 
streamline the data validation process, and examine the effect of data 
validation on program error rates over the course of the data 
validation time series information. The Department will conduct this 
analysis in the process of reauthorizing the data validation 
collection, including the methodology that is used, in the fall of2014. 

GAO Recommendation 2d: 
Regularly monitoring Social Policy Research Associates' corrections 
and analyses of state WIA participant data, sharing this information 
with states, and coordinating with states to ensure that any 
corrections are appropriate and accurate. 

The Department shares GAO's interest in ensuring that Social Policy 
Research Associates' (SPRA) analyses of state WIA participant data, 
including any corrections to the data, are appropriate and accurate. 
Thus the Department already monitors and shares these analyses. SPRA's 
corrections have been publicly available with the data set from the 
inception of WIA. Since Program Year 2011, the Department has provided 
SPRA's analyses and corrections to the states (through ETA Regional 
Offices) on a quarterly basis for states to either correct or dispute. 
This is a formal and recurring process that is "forward" working in 
the sense that any issues raised are to be addressed or corrected in
the next quarterly reporting cycle. In addition, ETA Regional Offices 
have begun to analyze state WIASRD data on a regular basis as a part 
of their annual review cycles. We are pleased with our progress over 
the past couple of years in obtaining states' feedback on SPRA's 
analyses. 

GAO Recommendation 2e: 
Collecting and disseminating promising practices on data collection 
and reporting to states and local areas on a regular basis. 

The Department agrees with GAO that "promising practices" in data 
collection and reporting should be shared with the public workforce 
system at the state and local levels. Data collection and reporting 
related topics are included in the "Workforce System Strategies" 
component of workforce 3 one, the Department's on-line interactive
communications and learning platform designed to build the capacity of 
the workforce investment system. Workforce Systems Strategies, 
launched in September 2012, represents the Department's effort to 
begin identifying a range of potential workforce strategies that are 
informed by research evidence or peer expertise. Users can receive
weekly e-alerts about the new resources posted to the site each week 
related to their chosen topics. This system of shared promising 
practices continues to expand with the addition of approximately 30 
resources per month. 

We agree that it is important to have reliable program data for 
effectively managing and evaluating programs, therefore, we encourage 
you to consider the Department's enclosed technical comments on the 
draft report. 

Again, thank you for the opportunity to review the draft report. If 
you would like additional information, please do not hesitate to call 
me at (202) 693-2700. 

Signed by: 

Eric M. Seleznow: 
Acting Assistant Secretary: 

[End of section] 

Appendix IV: GAO Contact and Staff Acknowledgments: 

GAO Contact: 

Revae E. Moran, (202) 512-7215 or moranr@gao.gov: 

Staff Acknowledgments: 

In addition to the contact named above, Meeta Engle, Assistant 
Director; Theodore Alexander; Jenn McDonald; and Brian Schwartz made 
key contributions to this report. Also contributing to this report 
were Jessica Botsford, David Chrisinger, Kathy Leslie, Mimi Nguyen, 
Carol Patey, Rhiannon Patterson, Catherine Roark, Jerry Sandau, Walter 
Vance, and Charles Youman. 

[End of section] 

Related GAO Products: 

Workforce Investment Act: Local Areas Face Challenges Helping 
Employers Fill Some Types of Skilled Jobs. [hyperlink, 
http://www.gao.gov/products/GAO-14-19]. Washington D.C.: December, 
2013. 

Workforce Investment Act: Additional Actions Would Further Improve the 
Workforce System. [hyperlink, 
http://www.gao.gov/products/GAO-07-105IT]. Washington, D.C.: June 28, 
2007. 

Workforce Investment Act: Employers Found One-Stop Centers Useful in 
Hiring Low-Skilled Workers; Performance Information Could Help Gauge 
Employer Involvement. [hyperlink, 
http://www.gao.gov/products/GAO-07-167]. Washington, D.C.: December 
22, 2006. 

Workforce Investment Act: Labor and States Have Taken Actions to 
Improve Data Quality, but Additional Steps Are Needed, [hyperlink, 
http://www.gao.gov/products/GAO-06-82]. Washington, D.C.: Nov. 14, 
2005. 

Workforce Investment Act: Labor Should Consider Alternative Approaches 
to Implement New Performance and Reporting Requirements. [hyperlink, 
http://www.gao.gov/products/GAO-05-539]. Washington, D.C.: May 27, 
2005. 

Workforce Investment Act: Substantial Funds Are Used for Training, but 
Little Is Known Nationally about Training Outcomes. [hyperlink, 
http://www.gao.gov/products/GAO-05-650]. Washington, D.C.: June 29, 
2005. 

Workforce Investment Act: Labor Actions Can Help States Improve 
Quality of Performance Outcome Data and Delivery of Youth Services. 
[hyperlink, http://www.gao.gov/products/GAO-04-308]. Washington, D.C.: 
February 23, 2004. 

Workforce Investment Act: States and Local Areas Have Developed 
Strategies to Assess Performance, but Labor Could Do More to Help. 
[hyperlink, http://www.gao.gov/products/GAO-04-657]. Washington, D.C.: 
June 1, 2004. 

Workforce Investment Act: Improvements Needed in Performance Measures 
to Provide a More Accurate Picture of WIA's Effectiveness. [hyperlink, 
http://www.gao.gov/products/GAO-02-275]. Washington, D.C.: February 1, 
2002. 

[End of section] 

Footnotes: 

[1] Pub. L. No. 105-220, 112 Stat. 936. GAO, Workforce Investment Act: 
Additional Actions Would Further Improve the Workforce System, 
[hyperlink, http://www.gao.gov/products/GAO-07-1051T] (Washington, 
D.C.: June 28, 2007); Workforce Investment Act: Labor and States Have 
Taken Actions to Improve Data Quality, but Additional Steps Are 
Needed, [hyperlink, http://www.gao.gov/products/GAO-06-82] 
(Washington, D.C.: Nov. 14, 2005). 

[2] Training and Employment Guidance Letter No. 36-11, Announcement of 
American Job Center Network (Washington, D.C.: June 14, 2012) strongly 
encouraged states and local areas to refer to the one-stop system as 
the American Job Center network and to one-stop career centers as 
American Job Centers, in order to increase job seeker and employer 
awareness of available workforce development resources. 

[3] These proposals include: Supporting Knowledge and Investing in 
Lifelong Skills Act (SKILLS Act), H.R. 803, 113th Cong. (2013); 
Workforce Investment Act of 2013, H.R. 798, 113th Cong. (2013); 
Workforce Investment Act of 2013, S. 1356,113th Cong. (2013); and 
Careers through Responsive, Efficient and Effective Retraining Act 
(CAREER Act) S.804, 113th Cong. (2013). 

[4] [hyperlink, http://www.gao.gov/products/GAO-07-1051T]; [hyperlink, 
http://www.gao.gov/products/GAO-06-82]. 

[5] U.S. Department of Labor, Office of Inspector General. Audit of 
Workforce Investment Act Data Validation for the Adult and Dislocated 
Worker Programs, 03-09-003-03-390 (Washington, D.C.: Sept. 30, 2009). 

[6] The scope of the request only covered two of the three WIA 
programs--the Adult and Dislocated Worker Programs; it did not cover 
the WIA Youth Program. 

[7] We used DOL data from program year 2010 to identify total federal 
spending on the Adult and Dislocated Worker Programs and the extent of 
data issues because they were the most recent data available when we 
selected our sample of states. Program year 2010 ran from July 1, 2010 
through June 30, 2011. Core "self-services" are services that can be 
provided without significant staff assistance. We conducted in-person 
interviews with DOL officials in Regions 1 (Boston), 3 (Atlanta), and 
5 (Chicago); state and local workforce officials in Massachusetts, 
Washington, Illinois, Georgia, and California; and American Job Center 
officials in Maryland. We conducted telephone interviews with DOL 
officials in Regions 2 (Philadelphia), 4 (Dallas), and 6 (Sacramento), 
and with state workforce officials in Maryland, Utah, and South 
Dakota. Because, in both South Dakota and Utah, the entire state is 
considered to be a single service delivery area, we did not interview 
American Job Center staff in either state. Our selection criteria 
included "core self-service" based on past findings by GAO and DOL's 
Office of Inspector General that this variable is often underreported, 
and thus likely to affect the number of reported participants in the 
WIA programs. 

[8] We analyzed data for program year 2011 because they were the most 
recent full year of WIASRD data available when we conducted our 
review. Program year 2011 ran from July 1 2011 through June 30 2012. 

[9] For this study, we define "WIA program participants" as 
individuals who have a valid Date of Program Participation for the WIA 
Adult or Dislocated Worker Programs and who have been counted and 
reported to DOL as WIA Adult or Dislocated Worker Program 
participants. Program year 2011 ran from July 1, 2011 through June 30, 
2012. WIASRD is a national database of individual records containing 
information on the characteristics, activities, and outcomes of all 
enrolled participants who received services or benefits under WIA. 
Participant data are typically collected by local staff at American 
Job Centers and entered into a state or local data system. After the 
state receives data from local areas, it compiles and formats the 
information and submits it electronically to DOL. 

[10] 29 U.S.C. § 2864(d)(4)(E). 

[11] A dislocated worker is an individual who: (1) has been terminated 
or laid off, or has received a notice of termination or layoff from 
employment, and is eligible for unemployment compensation, has 
exhausted unemployment compensation, or is not eligible for 
unemployment compensation due to insufficient earnings or having 
worked for a non-covered employer but has demonstrated an appropriate 
attachment to the workforce and is unlikely to return to a previous 
industry or occupation; (2) has been terminated or laid off or 
received notification of termination or layoff from employment as a 
result of a permanent closure or substantial layoff; (3) is employed 
at a facility where the employer has made the general announcement 
that the facility will close within 180 days; (4) was self employed 
but is unemployed as a result of general economic conditions in the 
community or because of a natural disaster; or (5) is a displaced 
homemaker. See 20 U.S.C. § 2801(9). 

[12] Appropriations are tracked by fiscal year but participants are 
tracked by program year. 

[13] 29 U.S.C. §§ 2831 and 2864(c)(2). 

[14] 29 U.S.C. § 2864(d)(2), (3) and (4). 

[15] According to DOL, the determination of need for training can 
itself be a core and/or intensive service, such as an assessment or 
development of an Individual Employment Plan. Training and Employment 
Guidance Letter No. 14-08, Guidance for Implementation of the 
Workforce Investment Act and Wagner-Peyser Act Funding in the American 
Recovery and Reinvestment Act of 2009 and State Planning Requirements 
for Program Year 2009 (Washington D.C.: Mar. 18, 2009). 

[16] 29 U.S.C. § 2871(b)(2)(A)(i). Training and Employment Guidance 
Letter No. 17-09, Quarterly Submission of Workforce Investment Act 
Standardized Record Data (WIASRD) (Washington, D.C.: Mar. 10, 2010). 

[17] DOL provides guidance to states in the form of Training and 
Employment Guidance Letters and Notices. Training and Employment 
Guidance Letter No. 27-10, Program Year 2010/Fiscal Year 2011 
Performance Reporting and Data Validation Timelines (Washington, D.C.: 
May. 11, 2011). Program years run from July 1 through June 30. 

[18] WIASRD also captures data on participants in the WIA Youth 
Program, but this is outside of the scope of our review. 

[19] In this section, we are referring specifically to DOL's data 
collection and reporting guidance for WIASRD. 

[20] GAO, Internal Control Management and Evaluation Tools, 
[hyperlink, http://www.gao.gov/products/GAO-01-1008G] (Washington, 
D.C.: August 2001), and Standards for Internal Control in the Federal 
Government, [hyperlink, 
http://www.gao.gov/products/GAO/AIMD-00-21.3.1] (Washington, D.C.: 
November 1999). 

[21] Used in this context, integration can refer to the co-location of 
services at an American Job Center, or to integrated funding to 
provide any given service. 

[22] Training and Employment Guidance Letter No. 17-05, Common 
Measures Policy for the Employment and Training Administration's (ETA) 
Performance Accountability System and Related Performance Issues 
(Washington, D.C.: Feb. 17, 2006). 

[23] These trends are independent of the amounts of federal funding to 
states for the Adult and Dislocated Worker Programs. 

[24] The Wagner-Peyser Program is one of the mandatory American Job 
Center partner programs. The Wagner-Peyser Act of 1933 established a 
nationwide system of public employment offices known as the Employment 
Service. The act was amended by the Workforce Investment Act of 1998 
to require that employment services be delivered through the one-stop 
services delivery system. The Employment Service focuses on providing 
a variety of core employment related labor exchange services including 
but not limited to job search assistance, job referral, and placement 
assistance for job seekers. 

[25] Data from the program year 2011 WIASRD was used to estimate the 
total number of participants in the WIA Adult Program. Estimates from 
the U.S. Census Bureau of population 18 years of age and over for 2011 
were used to rank each state and territory based on population. Our 
use of the word "states" refers to the 50 states plus Puerto Rico, the 
District of Columbia, and the U.S. Virgin Islands. 

[26] Regional office staff informed us that some states are still not 
reporting recipients of WIA-funded self-services as program 
participants. 

[27] DOL's Training and Employment Guidance Letter No. 17-05 defines 
the date of program participation, which refers to WIASRD variable 
302: Date of Program Participation. 

[28] We are referring to WIASRD variables 332: Date of First Staff-
assisted Core Service. 

[29] As noted in our background section, core services are a 
prerequisite for intensive and training services. 

[30] Specifically, officials from DOL's national office said that, 
when its contractor for WIASRD prepares the publicly available data 
files, for all participants without a date of first staff-assisted 
core service but with a date of first intensive service or training, 
the contractor will estimate the date of that core service by 
replacing the missing date with the Date of Program Participation. 

[31] Training and Employment Guidance Letter No. 17-09. We are 
referring to WIASRD variable 341: Type of Training Service #1. DOL's 
Training and Employment Guidance Letter No. 17-09 lists six types of 
training but does not provide detailed definitions for them. These 
categories include on-the-job training, skill upgrading and 
retraining, entrepreneurial training, ABE or ESL in combination with 
training, customized training, or other occupational skills training. 
Although "Customized training" and "on-the job training" are defined 
in statute ((29 U.S.C. §§ 2801(8) and 2801(31)), they are not defined 
in any of DOL's guidance. 

[32] Some core self-service and informational activities can be 
conducted remotely through internet connections to an American Job 
Center, satellite center, or partner agency. Participants who only 
receive core self-services are excluded from WIA performance outcome 
measures. 29 U.S.C. 2871(b)(2). Training and Employment No. 8-10, 
Workforce Investment Act Self-Service Participant Reporting What, 
Where, and How (Washington, D.C.: Aug. 26, 2010). 

[33] 20 U.S.C. § 2935(c)(2). 

[34] Two of these states, Georgia and California, were in the process 
of transitioning their state information system to a contractor when 
we interviewed state officials. 

[35] Training and Employment Guidance Letter No. 17-09. We are 
referring to WIASRD variable 331: Received Core Self-Services and 
Informational Activities. 

[36] Training and Employment Notice No. 8-10. 

[37] Training and Employment Notice No. 8-10. 

[38] Our analysis did not identify the reason that each of these 
states did not report data on self-service-only recipients. Some 
states may not be recording this data for reasons other than 
limitations in their information systems. 

[39] U.S. Department of Labor, Workforce Investment Act Standardized 
Record Data File Layout (Washington, D.C.). 

[40] WIA required the Governor of each state to reserve a portion of 
the funding for the WIA Adult, Dislocated Worker, and Youth Programs 
for certain statewide employment and training activities, which DOL 
officials told us can include investment in information systems. 29 
U.S.C. § 2853(a)(1). For 2011, 2012, and 2013, Congress reduced the 
maximum percentage states could set aside for these activities from 15 
percent to 5 percent. 

[41] [hyperlink, http://www.gao.gov/products/GAO-05-927], Managing for 
Results: Enhancing Agency Use Of Performance Information for 
Management Decision Making (Washington, D.C.: Sept. 9, 2005). 

[42] DOL has a contract with SPRA to correct and analyze WIASRD data 
and provide data files and reports on the accuracy of the data 
reported by states that are made available to the public via DOL's 
website. 

[43] Social Policy Research Associates, Workforce Investment Act 
Program Year 2010 Fourth-Quarter Data Quality Revisions (Oakland, 
C.A.: Nov, 7, 2011); Program Year 2011 Fourth-Quarter Data Quality 
Revisions (Oakland, C.A.: Nov. 28, 2012). 

[44] [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1]. 

[45] [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1]. 

[46] Training and Employment Guidance Letter No. 28-11.Program Year 
2011/Fiscal Year 2012 Performance Reporting and Data Validation 
Timelines (Washington D.C.: May 9, 2012). 

[47] [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1].  

[48] Data validation also includes report validation--assessing 
whether states' software accurately calculated required WIA 
performance measures. In response to a prior GAO recommendation, DOL 
established a 2 percent error rate threshold for report validation, 
which is tied to financial incentives and sanctions. Officials from 
one of DOL's regional offices stated that they noticed significant 
decreases in report validation error rates after DOL began using them 
to calculate financial awards. For more information on the 
recommendations, see: [hyperlink, 
http://www.gao.gov/products/GAO-06-82]; Training and Employment 
Guidance Letter No. 09-07, Revised Incentive and Sanction Policy for 
Workforce Investment Act Title IB Programs Washington, D.C.: Oct. 10, 
2007). 

[49] In accordance with WIA, DOL provides financial awards to states 
for performance that exceeds negotiated levels and assessed penalties 
for performance that does not meet negotiated performance levels. 20 
U.S.C. § 9273 and 29 U.S.C. §2871(g). 

[50] We are referring to WIASRD variable 303: Date of Exit. 

[51] We are referring to WIASRD variable 332: Date of First Staff 
Assisted Core Service. The error rates presented are "reported data 
error rates." Data element validation produces an overall estimate of 
the error rate for each data element that has been selected for 
validation. Because certain data elements may not be present in every 
sampled record, data validation also produces a reported data error 
rate, which includes in the denominator only those records for which 
the particular data element was validated. In this situation, the 
error rate equals the number of records in error divided by the total 
number of records for which the particular data element was validated, 
weighted to account for the over-and under-sampling of particular 
records. 

[52] [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1]. 

[53] [hyperlink, http://www.gao.gov/products/GAO-06-82]. According to 
DOL officials, these reviews occur every 2 to 3 years. 

[54] U.S. Department of Labor, Office of Inspector General, 03-09-003-
03-390; U.S. Department of Labor, Core Monitoring Guide (Washington, 
D.C.: April 2005); U.S. Department of Labor, Core Monitoring Guide: 
Financial Supplement (Washington, D.C.: October 2007); U.S. Department 
of Labor, Core Monitoring Guide: Data Validation Supplement 
(Washington, D.C.: March 2011). U.S. Department of Labor, Core 
Monitoring Guide with the Formula Grant Supplement (Washington, D.C.: 
April 2012); U.S. Department of Labor, Data Reporting and Validation 
System (DRVS): User Handbook for DRVS 7.2 (Washington, D.C.: April 
2009). 

[55] U.S. Department of Labor Employment and Training Administration. 
Fiscal and Performance Training Forum 2011: Data Validation--Looking 
Back and Moving Forward (Washington D.C.: June 17, 2011), 122. 

[56] [hyperlink, http://www.gao.gov/products/GAO-05-927]. 

[57] These programs include the Wagner-Peyser, WIA Adult, Dislocated 
Worker, and Youth, Jobs for Veterans State Grants, National Emergency 
Grants, and Trade Adjustment Assistance Programs. 

[58] Although DOL's guidance for reporting data in WIASRD encourages 
states and local areas to provide integrated services through multiple 
programs, each program has its own reporting requirements, according 
to officials from DOL's national office. As a result, it is not 
possible to track individual job seekers who receive services from 
multiple programs across the workforce system, or to determine the 
proportion of resources provided by each program for a particular 
service, or to attribute participant outcomes to those programs. 

[59] This would involve adding variables to WIASRD that would allow it 
to capture all the data required by any of the workforce programs 
included in WISPR. The modified WIASRD layout would utilize variable 
definitions identical to those of WISPR and is designed to facilitate 
full scale implementation of WISPR. 

[60] [hyperlink, http://www.gao.gov/products/GAO/PEMD-95-1], Program 
Evaluation: Improving the Flow of Information to the Congress 
(Washington D.C.: Jan. 30, 1995). 

[61] [hyperlink, http://www.gao.gov/products/GAO-13-570], Program 
Evaluation: Strategies to Facilitation Agencies' Use of Evaluation in 
Program Management and Policy Making (Washington D.C.: June 16, 2013). 

[62] In June 2012, DOL awarded nearly $147 million in Workforce 
Innovation Fund grants to states to develop and expand innovative 
strategies to help Americans return to work by delivering services 
more efficiently and facilitating cooperation across programs and 
funding streams. Grant awards ranged from about $1.4 million to $12 
million. In June 2013, DOL awarded $6.4 million in state grants for 
the third round of the Workforce Data Quality Initiative designed to 
improve the quality and availability of workforce data. Grant awards 
for that program ranged from about $700,000 to about $1.2 million. 

[63] For this study, we define "WIA program participants" as 
individuals who have a valid Date of Program Participation for the WIA 
Adult or Dislocated Worker Programs and who have been counted and 
reported to DOL as WIA Adult and Dislocated Worker Program 
participants. Participant data are typically collected by local staff 
at American Job Centers and entered into a state or local data system. 
After the state receives data from local areas, it compiles and 
formats the information and submits it electronically to DOL. 

[64] Program year 2011 runs from July 1, 2011 through June 30, 2012. 
WIASRD is a national database of individual records containing 
information on the characteristics, activities, and outcomes for all 
enrolled participants who received services or benefits under WIA. 

[65] We conducted these interviews between September 2012 and June 
2013. 

[66] We did not interview local officials in Utah and South Dakota 
because, for both states, the entire state is considered to be a 
single service delivery area and there are no designated local areas. 

[67] We used DOL data from program year 2010 to identify total federal 
spending on the Adult and Dislocated Worker Programs and the extent of 
data issues because they were the most recent data available when we 
selected our sample of states. Program year 2010 ran from July 1, 2010 
through June 30, 2011. Core "self-services" include services that can 
be provided without significant staff assistance. 

[68] Our analysis included all participants in the WIA Adult and 
Dislocated Worker Programs whose participation date was before July 1, 
2012 and who had either not exited the programs or exited after July 
1, 2011. We analyzed program year 2011 data because they were the most 
recent full year of WIASRD data available when we conducted our 
review. Program year 2011 ran from July 1, 2011 through June 30, 2012. 

[69] These reviews typically occur every 3 to 4 years. The date of the 
reviews we analyzed ranged from program year 2006 to program year 2011. 

[70] The error rates presented are "reported data error rates." Data 
element validation produces an overall estimate of the error rate for 
each data element that has been selected for validation. Because 
certain data elements may not be present in every sampled record, data 
validation also produces a reported data error rate, which includes in 
the denominator only those records for which the particular data 
element was validated. In this situation, the error rate equals the 
number of records in error divided by the total number of records for 
which the particular data element was validated, weighted to account 
for the over-and under-sampling of particular records. 

[71] We reviewed the results of 11 data elements from the Adult 
Program and 12 from the Dislocated Worker Program. Data elements 
included in both program are ProgramParticipationDate, 
FirstCoreServiceDate, FirstIntensiveService, NeedyFamilyStatus, 
DateEnterTraining, DateExitTraining, TrainingService1, VeteranStatus, 
OtherExitReasons, and ProgramExitDate. The Adult Program also included 
LowIncomeStatus and the Dislocated Worker Program also included 
DislocationDate and DisplacedHomemakerIndicator. 

[72] WIASRD is a national database maintained by DOL of individual 
records containing information on the characteristics, activities, and 
outcomes for all participants enrolled in WIA programs who received 
services or benefits under WIA. We analyzed program year 2011 data 
because they were the most recent full year of WIASRD data available 
when we conducted our review. Program year 2011 ran from July 1, 2011 
through June 30, 2012. 

[End of section] 

GAO’s Mission: 

The Government Accountability Office, the audit, evaluation, and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the 
performance and accountability of the federal government for the 
American people. GAO examines the use of public funds; evaluates 
federal programs and policies; and provides analyses, recommendations, 
and other assistance to help Congress make informed oversight, policy, 
and funding decisions. GAO’s commitment to good government is 
reflected in its core values of accountability, integrity, and 
reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO’s website [hyperlink, http://www.gao.gov]. Each 
weekday afternoon, GAO posts on its website newly released reports, 
testimony, and correspondence. To have GAO e-mail you a list of newly 
posted products, go to [hyperlink, http://www.gao.gov] and select 
“E-mail Updates.” 

Order by Phone: 

The price of each GAO publication reflects GAO’s actual cost of 
production and distribution and depends on the number of pages in the 
publication and whether the publication is printed in color or black 
and white. Pricing and ordering information is posted on GAO’s 
website, [hyperlink, http://www.gao.gov/ordering.htm]. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or 
TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card, 
MasterCard, Visa, check, or money order. Call for additional 
information. 

Connect with GAO: 

Connect with GAO on facebook, flickr, twitter, and YouTube.
Subscribe to our RSS Feeds or E mail Updates. Listen to our Podcasts.
Visit GAO on the web at [hyperlink, http://www.gao.gov]. 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 
Website: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]; 
E-mail: fraudnet@gao.gov; 
Automated answering system: (800) 424-5454 or (202) 512-7470. 

Congressional Relations: 

Katherine Siggerud, Managing Director, siggerudk@gao.gov: 
(202) 512-4400: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7125: 
Washington, DC 20548. 

Public Affairs: 

Chuck Young, Managing Director, youngc1@gao.gov: 
(202) 512-4800: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7149: 
Washington, DC 20548. 

[End of document]