This is the accessible text file for GAO report number GAO-02-275 entitled 'Workforce Investment Act: Improvements Needed in Performance Measures to Provide a More Accurate Picture of WIA's Effectiveness' which was released on February 1, 2002. This text file was formatted by the U.S. General Accounting Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. United States General Accounting Office: GAO: Report to Congressional Requesters: February 2002: Workforce Investment Act: Improvements Needed in Performance Measures to Provide a More Accurate Picture of WIA's Effectiveness: GAO-02-275: Contents: Letter: Results in Brief: Background: States and Localities Are Taking Action to Implement WIA Performance Measures but Face Challenges in Doing So: Performance Measures May Not Accurately Assess Performance of the Three WIA-Funded Programs: Existing Performance Measures Fail to Gauge Overall One-Stop Performance: Conclusions: Recommendations for Executive Action: Agency Comments: Appendix I: WIA Performance Measures for Adults, Dislocated Workers, and Older Youth: Appendix II: Comments from the Department of Labor: Appendix III: GAO Contacts and Staff Acknowledgments: GAO Contacts: Staff Acknowledgments: Related GAO Products: Tables: Table 1: Performance Measures for the Three WIA-Funded Programs: Table 2: WIA’s Mandatory Programs and Related Federal Agencies: Table 3: Comparing Wage Replacement Rates for Two Job Seekers: Figures: Figure 1: Timing of Data Collection by Quarter for Selected Performance Measures: Figure 2: Oregon’s Bean Counter: Figure 3: Number of Months it Takes to Receive UI Data for Job Placement: Figure 4: Comparison of Performance Measures for 17 One-Stop Programs: Abbreviations: GAO: General Accounting Office: GPRA: Government Performance and Results Act: JTPA: Job Training Partnership Act: MDA: Mississippi Development Authority: MIS: management information system: OMB: Office of Management and Budget: TEGL: Training and Employment Guidance Letter: UI: Unemployment Insurance: WIA: Workforce Investment Act: WRIS: Wage Record Interchange System: [End of section] United States General Accounting Office: Washington, DC 20548: February 1, 2002: The Honorable Edward M. Kennedy: Chairman: Committee on Health, Education, Labor and Pensions: United States Senate: The Honorable James M. Jeffords: United States Senate: The Congress passed the Workforce Investment Act (WIA) in 1998 to begin unifying a fragmented employment and training system and to better serve job seekers and employers. To create a more comprehensive workforce investment system, WIA requires states and localities to bring together most federally funded employment and training services into a single system, called the one-stop center system. Seventeen programs across four federal agencies—programs such as the Employment Service (Wagner-Peyser) and Adult Education and Literacy—must provide services through this one-stop system.[Footnote 1] Three of these programs, whose funding is authorized by WIA under Title I to provide services to adults, dislocated workers, and youth, replace those previously funded under the Job Training Partnership Act (JTPA). [Footnote 2] These three new WIA programs, authorized at about $3.7 billion in fiscal year 2001, have performance measures established under WIA that states and localities must track in order to demonstrate the effectiveness of the programs. These performance measures gauge program results in the areas of job placement, employment retention, and earnings change, as well as skill attainment and customer satisfaction. States are held accountable by the U.S. Department of Labor for their performance in these areas. If they fail to meet their expected performance levels, they may suffer financial sanctions; if they meet or exceed their levels, they may be eligible to receive additional funds, called incentive grants.[Footnote 3] In order to make objective decisions about which states receive incentives and sanctions, WIA seeks to ensure that states collect and report comparable performance data on all participants. In an effort to assess the effectiveness of WIA’s performance measures and whether they will yield useful information that can clearly demonstrate performance under WIA, we assessed (1) the progress states and localities have made and the issues they have faced in implementing performance measures for the three WIA-funded programs; (2) how useful the WIA performance measures are in accurately gauging the performance of the three WIA-funded programs; and (3) beyond gauging the performance of the three WIA-funded programs, how well the performance of the one-stop system is being measured. Our review is based on a survey of WIA program administrators in all 50 states; visits to Florida, Indiana, Missouri, Oregon, and Pennsylvania, where we interviewed state and local workforce investment officials; interviews with Labor officials and with national associations representing state and local workforce development officials; and a review of relevant documents. We selected the five states for site visits based on a variety of factors, including their experience using unemployment insurance (UI) wage record data to measure outcomes, the status of their data collection systems, and their progress in developing additional measures for state and local use. Our review focused on measures for Title I Adult, Dislocated Worker, and Older Youth programs. We did not review the performance measures for younger youth (age 14-18).[Footnote 4] We conducted our work between December 2000 and August 2001 in accordance with generally accepted government auditing standards. Results in Brief: States and localities have taken action to implement the new performance measurement system for the three WIA-funded programs but reported confronting a number of challenges in doing so. As part of their implementation efforts, states had to change the way they collected and reported performance data from the way they were collected for previous programs. Most states decided to develop new automated data systems, but 15 states do not have their systems completely in place. The timing of the guidance from Labor on how to report the data--not issued until well into the first year of implementation--delayed the development of these systems. States and localities also faced challenges in implementing these measures due to their complexity and the resource demands created by new measures; some had to develop new procedures to obtain access to sensitive records. For example, a new measure to assess customer satisfaction for job seekers and employers requires states to implement complicated procedures to conduct a customer satisfaction survey. In addition, state officials expressed concern that the levels of performance that they are expected to achieve are too high. Because the program was in its first year, there were little or no data that could be used as a point of comparison, or baseline, for establishing performance levels. Furthermore, state and local officials believed that the established performance levels did not take into account local economic conditions. The performance levels are of particular concern to state and local officials because failure to meet them can result in financial sanctions. As a result, states may be choosing to serve only those job seekers who are most likely to be successful. For example, to meet their performance levels to replace earnings for workers who have been dislocated from their jobs, local staff might provide WIA services only to job seekers who have the best prospect of finding work that can match or surpass their previous earnings. Even when fully implemented, WIA performance measures may still not provide a true picture of WIA-funded program performance largely because data are not comparable across states or timely. The measures include many of the indicators relevant to an employment and training program, such as getting and keeping jobs and increasing wages and skills. However, data may not accurately gauge performance and are not comparable across states for a variety of reasons. For example, Labor’s guidance on enrolling job seekers—a process called registration—does not provide clear direction on when to start collecting performance data on participants, and we found that states and localities differed on whom they tracked and when. In addition, UI wage records, required in order to track performance outcomes, are not easily used to identify employment across state lines, and some states are more successful than others in obtaining this information. Only seven states are currently using a clearinghouse—called the Wage Record Interchange System (WRIS)—that allows states to exchange UI data. Many of the remaining states are reluctant to participate, often citing cost concerns. In addition, the time lag associated with UI data becoming available (as much as 9 months or longer) results in states not being able to use the data to manage their programs in the short term. While there are measures designed to gauge the performance of the three WIA-funded programs, there are no measures to gauge the performance of the one-stop system as a whole. At least 17 programs provide services through the one-stop system and most have their own performance measures. Although these performance measures may be used for assessing outcomes for individual programs, they cannot be used to measure the success of the overall system. For example, no program has a measure to track job seekers who use only self-service or informational activities offered through the one-stop, which may constitute a large proportion of job seekers. Not knowing how many job seekers use the one-stop’s services limits the one-stop’s ability to assess its impact. Furthermore, state and local officials told us that having multiple performance measures has impeded coordination among programs. There has been limited progress in developing overall performance measures for the one-stop system. Labor convened a working group to develop additional indicators of the one-stop system’s performance, but they have not yet issued them. To provide the Congress and the public with a more accurate picture of WIA performance, we are making recommendations to Labor in this report to postpone the implementation of financial sanctions, to expedite the development of criteria for renegotiating performance levels, to more clearly define policies and measures, and to develop ways to address the challenges in using UI data. In its written comments, Labor concurred with our findings and recommendations. Background: Labor required states to implement major provisions of WIA Title I by July 1, 2000, although some states began implementing provisions of WIA as early as July 1999. Services provided under WIA represent a marked change from those provided under the previous program, allowing for a greater array of services to the general public. WIA is designed to provide for greater accountability over what existed previously: it established new performance measures and a new requirement to use UI data to track and report on the achievements of the three WIA-funded programs. WIA also requires that many federal programs work together to provide employment and training services through the one-stop system. WIA-Funded Services Represent a Change from those Funded under JTPA: Program services provided under WIA represent a marked change from those provided under JTPA. When WIA was enacted in 1998, it replaced the JTPA programs for economically disadvantaged adults and youth and for dislocated workers with three new programs—WIA Adult, Dislocated Worker, and Youth—that provide a broader range of services to the general public, no longer using income to determine eligibility for all program services.[Footnote 5] The newly authorized WIA programs [Footnote 6] no longer focus exclusively on training but provide for three tiers, or levels, of service for adults and dislocated workers: core, intensive, and training. Core services include basic services such as job searches and labor market information. These activities may be self-service or require some staff assistance. Intensive services include such activities as comprehensive assessment and case management—activities that require greater staff involvement. Training services include such activities as occupational skills or on-the-job training. These tiers of WIA-funded services are provided sequentially. That is, in order to receive intensive services, job seekers must first receive at least one core service; to receive training services, a job seeker must first receive at least one core service and then at least one intensive service. Key to moving from core to a higher level of services is that the services are needed to help job seekers become self-sufficient. Labor’s guidance provides for monitoring and tracking to begin when job seekers receive core services that require significant staff assistance. Job seekers who receive core services that are self-service in nature are not included in the performance measures. WIA Performance Measures Are Designed to Increase Accountability for Three WIA-Funded Programs: WIA is designed to provide for greater accountability than the accountability provided for under JTPA. It does so by establishing new performance measures and a new requirement to use UI data to track and report on the achievements of the three WIA-funded programs. According to Labor, performance data collected from the states in support of the measures are intended to be comparable across states in order to maintain objectivity in determining incentives and sanctions. They are also intended to provide information to support Labor’s performance goals under the Government Performance and Results Act (GPRA)[Footnote 7] and for program evaluation. Some of the measures that relate to adults, dislocated workers, and older youth are similar to those used under JTPA, including job placement, job retention, and wage gains or replacement. Attainment of a credential[Footnote 8]—a degree or certification of skills or training completed—and customer satisfaction for both job seekers and employers are new under WIA. (See table 1 for a complete list of the WIA performance measures and appendix I for a more complete explanation of the performance measures discussed in this report.) Table 1: Performance Measures for the Three WIA-Funded Programs: WIA funding stream: Adult; Performance measure: 1. Entered employment rate; 2. Employment retention rate at 6 months; 3. Average earnings change in 6 months; 4. Entered employment and credential rate. WIA funding stream: Dislocated worker; Performance measure: 5. Entered employment rate; 6. Employment retention rate at 6 months; 7. Earnings replacement rate in 6 months; 8. Entered employment and credential rate. WIA funding stream: Older youth (age 19-21); Performance measure: 9. Entered employment rate; 10. Employment retention rate at 6 months; 11. Average earnings change in 6 months; 12. Entered employment/education/training and credential rate. WIA funding stream: Younger youth (age 14-18); Performance measure: 13. Skill attainment rate; 14. Diploma or equivalent attainment; 15. Placement and retention rate. WIA funding stream: Customer satisfaction; Performance measure: 16. Customer satisfaction for participants; 17. Customer satisfaction for employers. Source: U.S. Department of Labor Employment and Training Administration, Training and Employment Guidance Letter No. 7-99 (Mar. 3, 2000). [End of table] In contrast to JTPA, for which data on outcomes were obtained through follow-ups with job seekers, WIA requires states to use UI wage records to track outcomes. According to Labor’s guidance, if a program participant does not appear in the UI wage records, states may use supplemental data sources, such as follow-ups with participants and employers, to track entered employment, retention, and credential attainment. However, only UI wage records may be used to calculate earnings change and replacement. Unlike JTPA, which established expected performance levels using a computer model, WIA requires states to negotiate with Labor to establish expected performance levels for each measure. States, in turn, must negotiate performance levels with each local area. The law requires that these negotiations take into account differences in economic conditions, participant characteristics, and services provided. To derive equitable performance levels, Labor and the states use historical data to develop their estimates of expected performance levels. These estimates provide the basis for negotiations. WIA holds states accountable for achieving their performance levels by tying those levels to financial sanctions and incentive funding. States that meet their performance levels under WIA are eligible to receive incentive grants that may generally range from $750,000 to $3 million. [Footnote 9] To be eligible for an incentive grant, states must also meet levels established under the Department of Education’s Vocational Education (Perkins Act) and Adult Education and Literacy programs. States that do not meet their performance levels under WIA are subject to sanctions.[Footnote 10] If a state fails to meet its performance levels for 1 year, Labor provides technical assistance, if requested. If a state fails to meet its performance levels for 2 consecutive years, it may be subject to up to a 5-percent reduction in its annual WIA formula grant. Under JTPA, the most stringent sanction was the possible reorganization of the local service delivery organization. WIA Requires that Many Federal Programs Work Together to Provide Services through the One-Stop System: In addition to establishing the three new programs, WIA requires that states use the one-stop center system to provide services for these and many other employment and training programs. This system was developed by states prior to WIA through One-Stop Planning and Implementation Grants from Labor. About 17 programs funded through four federal agencies are now required to provide services through the one-stop center under WIA.[Footnote 11] Table 2 shows the programs that WIA requires to provide services through the one-stop centers (termed mandatory programs) and the related federal agency. Table 2. WIA’s Mandatory Programs and Related Federal Agencies: Federal agency: Department of Labor; Mandatory programs: WIA adult; WIA dislocated worker; WIA youth; Employment Service (Wagner-Peyser); Trade adjustment assistance programs; Veterans’ employment and training programs; Unemployment Insurance; Job Corps; Welfare-to-Work grant-funded programs; Senior Community Service Employment Program; Employment and training for migrant and seasonal farm workers; Employment and training for Native Americans. Federal agency: Department of Education; Mandatory programs: Vocational Rehabilitation Program; Adult Education and Literacy; Vocational Education (Perkins Act). Federal agency: Department of Health and Human Services; Mandatory programs: Community Block Grant Services. Federal agency: Department of Housing and Urban Development (HUD) Mandatory programs: HUD-administered employment and training. [End of table] Under WIA, employers are expected to play a key role in establishing regional workforce development policies, deciding how services should be provided in the one-stop, and overseeing one-stop operations. Employers, who are encouraged to use the one-stop system to fill their job vacancies, are also seen as key one-stop customers under WIA. States and Localities Are Taking Action to Implement WIA Performance Measures but Face Challenges in Doing So: States and localities are taking action to implement performance measures for the three WIA-funded programs, but they have confronted several challenges in doing so. To implement these measures, states and localities had to change the way they collected and reported performance data. Most states we surveyed had to create new automated data systems to collect and report WIA data. Many state systems, however, are still not completely in place. The lack of final guidance from Labor on how to report the data slowed the development of these systems. States and localities also faced challenges in implementing the measures due to their complexity and the resource demands they created, and some had to develop new procedures to obtain UI wage records. In addition, states faced a new negotiation process with Labor to set performance levels for each measure. Many states believe these levels are too high because little or no baseline data were used, and the negotiations did not sufficiently account for differences in economic conditions and populations served. Under WIA, performance levels are now tied to incentives and sanctions so that states can be financially rewarded if they meet them or penalized if they do not. States reported that the need to meet these performance levels may lead local staff to focus WIA-funded services on job seekers who are most likely to succeed in their job search or who are most able to make wage gains. Management Information Systems Are Still Being Developed to Collect and Report WIA Data: As part of implementing WIA performance measures, states had to develop automated data systems to track the activities of individual WIA participants and report on performance.[Footnote 12] Based on our survey, most states developed a new automated data system, or management information system (MIS), to collect and report WIA performance data at the state level. The remaining states adapted their previous data collection systems used under JTPA. However, 15 states, regardless of whether they were developing a new system or adapting their existing system, reported that, as of August 2001, they did not have their system completely in place. All states expect to have completed their systems by July 2002. In some states, local areas do not use the state MIS to collect local WIA performance information. In these states, local areas must develop their own systems, taking time and resources to do so. The lack of timely reporting guidance slowed the development of the data systems. Final guidance on how states must report their performance to Labor was issued to the states in March 2001—8 months after the states were required to implement major provisions of WIA and begin collecting data.[Footnote 13] Lack of final guidance resulted in delays and costly program changes as states and local areas developed and adjusted their final systems. For example, one local area we visited decided to continue using its old system and delayed the development of a new system pending final guidance because it would be too costly and time-consuming to develop a system that might need to be changed. All states, regardless of whether or not they had implemented a new system, had to make changes in their automated MIS systems to accommodate the final guidance. Complex Performance Measures Stymied State and Local Efforts to Implement Them: States and localities reported that the complexity of WIA’s new performance measures made them difficult and time-consuming to implement. Many of the states that we surveyed commented that the measures were hard to follow because the calculations for the measures are complex and sometimes confusing, specifically who to include in the measures, when to collect the data for the measures, and how to calculate the measures.[Footnote 14] * Knowing who to include in the measures. It is difficult to know whether or not a job seeker should be counted in the measure for a program, even if he or she is served by the program. For example, a participant in the adult program who is already employed must be included in the retention and wage gain measures but cannot be counted in the entered employment measure. Yet, for the dislocated worker program, the entered employment measure can include those who may be employed when they enter the program.[Footnote 15] * Knowing when to measure performance for participants. The data for different performance measures can be collected in different quarters of the program year. For example, for customer satisfaction, data can be collected at two points in time, depending on how the participant exited the program (see figure 1). Data on entered employment for participants in the adult program are collected in the third quarter after exit; retention data are collected in the fifth quarter after exit.[Footnote 16] Data on earnings change and replacement are collected at two points in time: pre-program earnings data can be collected at registration, and post-program earnings data are collected in the fifth quarter after exit.[Footnote 17] If a program participant does not appear in the UI wage records, local staff can collect supplemental data to establish employment for the participant, but this must be recorded within 30 days after the WIA participant is found missing from the wage records. For entered employment, staff can collect supplemental data in the fourth quarter after a participant leaves (or exits) the program, but for retention, staff can collect it in the sixth quarter after exit. Supplemental data cannot be used to measure earnings change and replacement. Because the timing of data collection is complex and can be confusing, one local area in Oregon developed a tool it calls the “bean counter” to help local staff determine when to follow-up with participants so their performance counts in the calculations (see figure 2). Figure 1: Timing of Data Collection by Quarter for Selected Performance Measures: [Refer to PDF for image] This figure illustrates the following timeline: Program Year 1999 Quarter 3, Jan.-March: Participant registers in WIA; Measure: Earnings change/replacement: Collect data. Program Year 1999 Quarter 4, April-June: Program Year 2000 Quarter 1, July-September: Participant exits WIA; Program Year 2000 Quarter 2, October-December: Measure: Customer satisfaction: Collect data. Program Year 2000 Quarter 3, Jan.-March: Measure: Customer satisfaction: Collect data. Measure: Entered employment: Collect data. Program Year 2000 Quarter 4, April-June: Measure: Entered employment: 30 days to get supplemental data. Program Year 2001 Quarter 1, July-September: Measure: Retention: Collect data. Measure: Earnings change/replacement: Collect data. Program Year 2001 Quarter 2, October-December: Measure: Retention: 30 days to get supplemental data. Source: U.S. Department of Labor TEGL 7-99. [End of figure] Figure 2: Oregon’s Bean Counter: [Refer to PDF for image] This figure is an illustration of the Oregon Bean Counter. By sliding the paper card in the sleeve, staff can determine the appropriate quarter in which to follow up with participants. Source: Worksystems Inc. [End of figure] * Knowing how to calculate the measures. In order to calculate the measures, states must account for a variety of factors. The type and combination of these factors determine the calculations that will be used. For example, in calculating some of the measures for the adult program, states must consider (1) whether the job seeker is employed at registration, (2) whether he or she is employed at both the first and third quarters after exit, and (3) whether the data source used to confirm employment was UI records or supplemental data. This information in various combinations results in 14 different ways that adult participants can be grouped in order to calculate the measures. In addition to noting the complexity of the measures, state and local officials said that the new measures taxed their resources. States had to develop procedures to collect data for the new customer satisfaction measure in compliance with detailed guidance from Labor. The guidance calls for states to conduct a telephone survey from a random sample large enough to obtain 500 completed surveys from both participants and employers. Because this guidance changed over time, states had to revise their procedures accordingly. For example, a revision to the guidance issued in October 2001 required states to maintain an up-to- date list of participants’ names and addresses from which to sample—a requirement that was originally voluntary. One indication of states’ progress in implementing these measures may be reflected in their ability to submit complete quarterly reports. [Footnote 18] Quarterly reports require data on all 17 performance measures. For the quarterly report that was due in May 2001, all states submitted their reports, but, according to Labor, only 16 states were able to provide data for all 17 performance measures. For the quarterly report that was due in August 2001—more than 1 year after WIA implementation—all states submitted their reports, but only 23 could provide data on employer customer satisfaction.[Footnote 19] According to Labor, states could not fully report on customer satisfaction because they have not yet fully implemented procedures to measure it. One state had to compile the data manually because its MIS was not fully operational. States Adopt New Procedures to Access Sensitive Information: WIA’s new requirement that states use UI wage records to measure outcomes has led states to adopt new procedures to access these and other sensitive records. Unlike JTPA, which relied on surveys of participants to collect information on employment and earnings, WIA requires UI wage records to be used as the primary data source of employment and wage information—and the only data source for some measures, according to Labor’s guidance. To obtain employment and earnings information, states match information collected on individual WIA participants against state UI wage records. To access UI data from the state agency that oversees the UI database, some states had to establish data-sharing agreements. In Mississippi, for example, the agency responsible for overseeing WIA—the Mississippi Development Authority (MDA)—had to make arrangements with the agency that oversees the UI data—the Mississippi Employment Security Commission—to have them match the wage records and provide the results to MDA. In addition, some states may be more rigorous in protecting the confidentiality of UI records through privacy laws, which may add obstacles to collecting performance data. For example, Oregon law prohibits the release of WIA participants’ records without informed consent. Consequently, program providers had to enter into an agreement that established a protocol for collecting and sharing the data—one that developed safeguards to protect confidentiality. In addition, the state had to develop a process to ensure that WIA participants consented to the use of their protected records in this way. States Believe Performance Levels Are Set Too High: All the states we visited believed that some of the established performance levels for their measures were set too high for them to meet—either because they were set in absence of historical or baseline data or because negotiations did not sufficiently account for variations in economic conditions or population served. States reported that limitations in available baseline data made it difficult to set fair, realistic performance levels. The new measures on credentials and customer satisfaction, for instance, had no prior data available on which to set performance levels. Where baseline data were available, such as for the wage-related measures, the data were collected under JTPA, a program whose goals were different from those of WIA.[Footnote 20] In addition, some states believe that the performance levels did not account for variations in economic conditions, such as the slow growth in new or existing businesses that some areas have experienced. Performance levels also did not account for the many economically disadvantaged or hard-to-serve individuals seeking services in some local areas. States Report that Performance Levels May Determine Who Receives WIA- Funded Services: Many states reported that the need to meet performance levels may be the driving factor in deciding who receives WIA-funded services at the local level. All the states we visited told us that local areas are not registering many WIA participants, largely attributing the low number of WIA participants to concerns by local staff about meeting performance levels. Local staff are reluctant to provide WIA-funded services to job seekers who may be less likely to get and keep a job. One state official described how local areas were carefully screening potential participants and holding meetings to decide whether to register them. As a result, individuals who are eligible for and may benefit from WIA- funded services may not be receiving services that are tracked under WIA. Performance levels for the measures that track earnings change for adults and earnings replacement for dislocated workers may be especially problematic. Several state officials reported that local staff were reluctant to register already employed adults or dislocated workers. Officials in one state reported that some local areas had not yet registered any dislocated workers. State and local officials explained that it would be hard to increase the earnings of adults who are already employed or replace the wages of dislocated workers, who are often laid off from high-paying, low-skilled jobs or from jobs that required skills that are now obsolete. In addition, for dislocated workers, employers may provide severance pay or workers might work overtime prior to a plant closure, increasing these workers’ earnings before they are dislocated. As a result, many dislocated workers who come to the one-stop center have earned high wages just prior to being dislocated, making it hard to replace—let alone increase— their earnings. If high wages are earned before dislocation and lower wages are earned after job placement through WIA, the wage change will be negative, depressing the wage replacement level. As a result, a local area may not meet its performance level for this measure, discouraging service to those who may need it. A hypothetical example involving two workers dislocated at the same time illustrates this point (see table 3). One worker is a sales clerk with limited skills earning $25,000, the other a long-time factory worker with obsolete skills earning $60,000. Both are laid off from work and go to their local one-stop center seeking job placement assistance. The clerk is placed in a new job as a receptionist paying $25,000. By calculating his wage replacement from his salary as a clerk, the one-stop can claim a wage replacement rate of 100 percent. The factory worker eventually gets a job as a security guard earning $30,000, netting a wage replacement rate of 50 percent. As this example shows, a one-stop center can meet its performance levels more easily by serving the clerk than by serving the factory worker even though both job seekers may need the one-stop system’s resources to find a job or enhance their skills. Table 3: Comparing Wage Replacement Rates for Two Job Seekers: Scenario 1: Sales Clerk; Sales clerk earning $25,000; * Gets laid off; * Cannot find a job; * Comes to one-stop center for job placement assistance placement assistance; * The center gets him placed as a receptionist making $25,000; Wage replacement rate: $25,000/$25,000 = 100%. Scenario 2: Factory Worker; Factory worker earning $60,000 after 20 years of employment; * Gets laid off; * Cannot find a job; * Comes to one-stop center for job placement assistance placement assistance; • The center gets him placed as a security guard earning $30,000; Wage replacement rate: $30,000/$60,000 = 50%. Note: Example uses annual full-time earnings. When replacement rates are actually calculated using UI wage records, they will be based on earnings reported on a quarterly basis. [End of table] Some states and Labor are making efforts to address this disincentive to serve certain job seekers. Indiana instituted a policy allowing local areas to adjust their dislocated worker wage replacement rate in light of the significant dislocations they are facing.[Footnote 21] Texas uses a regression model to establish local performance levels that adjust for differences in factors, such as economic conditions and the characteristics of individuals served. Without this policy, said a Texas official, WIA programs would have registered fewer workers. Similarly, Michigan substantially reduced the penalties to local areas for failing to meet performance levels and found that the number of registered participants increased as a result of instituting less threatening sanctions. WIA requires that states be allowed to renegotiate their performance levels based on unanticipated circumstances. Labor is currently developing criteria that states can use to renegotiate their performance levels based on unanticipated circumstances, such as changes in economic conditions due to plant closings or shifts in unemployment for the current and future years. The guidance is expected to be released soon. Performance Measures May Not Accurately Assess Performance of the Three WIA-Funded Programs: Even when fully implemented, WIA performance measures may still not provide an accurate picture of performance for the three WIA-funded programs largely because data are neither comparable across states nor timely. State and local officials generally supported many of the performance measures as relevant indicators of the success of an employment and training program. However, the performance data collected and reported by states and localities are not comparable largely because of the lack of clear guidance on when to collect and report performance data and what constitutes a credential—the attainment of a certified skill or degree. In addition, while UI wage records are one of the best available sources of employment and earnings data, limitations in the data may hinder the ability of states and local areas to meet their performance levels and use the measures for short-term program management. Performance Measures Include Relevant Indicators, but Lack of Clear Guidance Affects Accuracy and Comparability of Performance Data: State and local officials in the states we visited generally support many of the performance measures as relevant indicators of the success of an employment and training program. For example, several officials cited the wage-related measures, such as job placement, retention, and earnings change, as important indicators of a successful employment and training program. The measures are also generally consistent with the goal of WIA to help individuals get and keep jobs and increase their wages and skills. In addition, the states noted that the measures provide a good basis for long-term evaluation. However, the performance data collected and reported by states and localities are not comparable—a critical component in creating a level playing field from which states’ relative performance can be evaluated. While there are various reasons that performance data are not comparable, one of the chief reasons is the lack of clear guidance for collecting and reporting performance data on participants. Labor has provided detailed written guidance to states on who should be registered under WIA and when this registration should occur, but the guidance is open to interpretation in some areas.[Footnote 22] The lack of a uniform understanding of when registration occurs and thus who should be counted toward the measures raises questions about both the accuracy and comparability of states’ performance data. For example, the guidance tells states to register adults and dislocated workers who need significant staff assistance designed to help with job seeking or acquiring occupational skills, but the state can decide what constitutes significant staff assistance.[Footnote 23] The guidance provides examples of when to register job seekers, but it sometimes requires staff to make subtle and subjective distinctions. For example, those who receive initial assessment of skill levels and the need for supportive services are not to be registered; those requiring comprehensive assessment or staff-assisted job search and placement assistance, including career counseling, are to be registered. In another example involving the classification of workshops, job seekers who participate are to be registered in some cases, but not in others. Labor has allowed states and local areas flexibility in implementing the registration policy, and we found that local areas differed on when they registered WIA job seekers. In one local area we visited, the one- stop center registers most job seekers who come into the center, even if staff assistance is minimal. At this center, a general orientation is sufficient for the job seeker to be registered under WIA. In contrast, another center in the same state registers only those job seekers who require significant staff assistance and are likely to benefit from intensive services. Similar disparities occurred in other states we visited. Labor has said there is little consistency across states in registering participants and has convened a work group to develop additional guidance on registration, but as yet, the issues remain unresolved. The lack of a definition for the credential measure is also leading to performance data that are not accurate or comparable across states. Labor allows the states and local areas to determine what constitutes a credential and to develop a statewide list of approved credentials with input from employers. Because states and, in many cases, local areas must define what constitutes a credential, what is currently counted as a credential differs within and across states. Some states may strictly define credentials to include only diplomas from accredited institutions or use only formal training completion criteria as defined by education partners. Other states may expand their criteria to count a broad variety of credentials, such as job readiness, on-the-job experience, and completion of workshops. Labor officials note that states’ performance levels for the credential measure are negotiated to take state and local definitions into account, and the measure is intended to help local employers gauge the readiness and skill level of job seekers. Nevertheless, given the broad range of definitions states and localities employ, the outcomes on the credential measure may be of limited value, even within a single state. Challenges in Using UI Wage Record Data Also May Affect the Accuracy and Comparability of Performance Data: UI wage records are one of the best available data sources for tracking the employment and earnings of individuals—a significant improvement over the less objective self-reporting methods of JTPA—but the limitations of the database pose challenges that need to be addressed. These challenges, if unresolved, may hinder states’ ability to meet their performance levels. As we have reported in prior work,[Footnote 24] one such limitation is that UI wage records, while covering about 94 percent of workers, exclude certain employment categories, such as self-employed persons, most independent contractors, military personnel, federal government workers, and postal workers. States, therefore, must develop alternative methods to track WIA participants who are employed in these uncovered occupations. Pennsylvania, for example, developed a partnership with other states in its region to share the cost of purchasing the rights to federal civil service and military personnel data. And Florida has developed agreements with the Department of Defense, the Office of Personnel Management, and the U.S. Postal Service to access employment and wage information on an annual basis. Our survey data indicate that 33 states are using additional or supplemental data to compensate for uncovered occupations, with only 27 of those using the supplemental data to count towards their performance levels. Thus, at least 23 states have not used additional data to help them meet their performance levels. Another limitation is that state UI databases include only wage record information on job seekers who get jobs within their state; they do not track job seekers who find jobs in other states. States cannot readily access UI wage records from other states to track outcomes under WIA, making it difficult to track individuals who receive services in one state but get a job in another. Over one-third of all of the states we surveyed reported that an estimated 16 to 30 percent of cases are not being picked up by their state’s UI wage record system. To fill in these gaps, seven states have agreements with other states—often those that share a common border—to exchange UI information. Indiana, for example, established an agreement with Illinois to trade data. If data are missing on particular participants, Illinois sends the cases to Indiana to see if the Indiana UI wage records have information on the job seeker. The value of these agreements, however, may be limited because job seekers may find work in a state that does not have an agreement with the one in which they received services. Another way to obtain UI data on workers who are employed out of state is through WRIS, a clearinghouse that makes UI wage records available to states seeking employment and wage information on their WIA participants. This information can provide outcome data on WIA participants to help states meet their performance levels. While WRIS was available for states to use by July 2001, only 7 out of the 50 states are currently able and ready to participate, with 8 others in various stages of completing the requirements for participation. [Footnote 25] Although many states have shown an interest in a system such as WRIS, many are reluctant to participate because Labor, while agreeing to cover all the costs of operating WRIS for its first year, has not yet agreed to pay for subsequent years. The estimated total cost of operating WRIS is $2 million annually, but states have not been given a definitive answer about how much it would cost them to participate after this first year if Labor does not continue funding. [Footnote 26] Because of this uncertainty regarding future costs, states are hesitant to commit to participation in WRIS. If not all states participate, the value of WRIS will be diminished—even for participating states—because no data will be available from nonparticipating states’ UI wage records. Lack of Timely Data Limits Use of Performance Measures for Short-Term Program Management: The lack of timely data, due to the time lag in obtaining UI wage records, makes it difficult for state and local officials to use the performance measures for short-term program management—because, for the wage-related measures, current available data on the measures will reflect performance from the previous program year. While UI wage records are the best available data source for documenting employment, the data collection and reporting process is slow and time-consuming. Data are generally collected from employers only once every quarter, and employers have 30 days after the quarter ends to report the data. In many states, employers—especially small businesses—are allowed to submit data in paper format, which then must be converted to electronic media. After data entry, information must be checked for errors and corrected. All of these steps take time. As a result, WIA program administrators are unable to get a timely picture of program performance. For example, we asked states in our survey how quickly job placement outcome data would be available to them from UI wage records. On the basis of our survey, we found that for 30 states, the earliest time period that job placement data would be available is 6 months or more after an individual entered employment, with 15 states reporting that it may take 9 months or longer. Similarly, for the employment retention measure, over half of states report that obtaining this information could take a year or longer. (See figure 3.) Figure 3: Number of Months it Takes to Receive UI Data for Job Placement: [Refer to PDF for image] This figure is a vertical bar graph depicting the following data: Shortest time period to receive UI job placement data: 3 months: 4 states; 4 months: 10 states; 5 months: 6 states; 6 months: 19 states; 7 months: 6 states; 8 months: 3 states; 9 months: 1 state; 11 months: 1 state. Longest time period to receive UI job placement data: 4 months: 1 state; 5 months: 7 states; 6 months: 3 states; 7 months: 9 states; 8 months: 15 states; 9 months: 6 states; 10 months: 2 states; 11 months: 2 states; 12 months: 1 state; 13 months: 2 states; 14+ months: 2 states. Note: Our survey asked two questions regarding UI time frames. One question asked for the earliest date states would expect to receive UI data; the other asked for the latest date. Source: GAO survey. [End of figure] The time delay in receiving UI wage record data makes it difficult for state and local officials to use the performance measures to gauge the effectiveness of their services. States report that not being able to get performance results in the same program year is a problem: it makes it difficult to manage programs and improve one-stop services. Labor reports that the performance measures are not intended to be a management tool. State and local officials, therefore, must develop alternative methods if they want to assess the quality of their services so they can identify problems and improve programs in a timely way. Labor has encouraged these efforts, but, while some local areas are finding ways to collect data to help them manage their programs, there is no cohesive effort at the federal level to share strategies and promising approaches for the Adult and Dislocated Worker Programs. [Footnote 27] Existing Performance Measures Fail to Gauge Overall One-Stop Performance: Although there are performance measures for the three WIA-funded programs and most of the programs required to provide services in the one-stop, no measures exist to assess how well the overall one-stop systems are working. The success of the one-stop system as a whole is not captured by the program measures of individual one-stop partners. Furthermore, combining the performance measures from mandatory programs does not provide a comprehensive picture of one-stop performance. Even when measures appear the same, comparing them is difficult because of differences in definitions and calculations. Beyond failing to provide a complete evaluation of one-stop performance, state officials reported that the separate reporting requirements of the partner programs have hampered coordination within their one-stop systems. While WIA did not establish any comprehensive measures to assess the overall one-stop system, it required that Labor take the lead in developing optional measures to help states assess progress toward their workforce investment goals. Labor has made limited progress on such performance measures, and only a few states have developed their own overall measures. Existing Performance Measures Fail to Capture Important One-Stop Features: The existing performance measures for participating one-stop programs fail to capture important one-stop features. First, it is difficult to get an unduplicated count of job seekers using the one-stop. While an individual may have multiple outcome measures for the services received through each of the programs at a one-stop, there is no single outcome measure for multiple services. In addition, separate reporting systems for each of the programs make it difficult to disaggregate data and track an individual’s outcome for those receiving multiple services. Second, other important aspects of one-stop performance are not included within the existing measures. Customer satisfaction measures used in support of WIA-funded programs and the Employment Service fail to measure how job seekers and employers believe they are being served by the one-stop system as a whole. Instead, these measures show satisfaction with the individual programs. Employer satisfaction is important under WIA because WIA created a more private-sector driven system. Capturing customer satisfaction of the system as a whole would reflect whether job seekers are successful at attaining the services they need to get jobs and would assess whether employers are satisfied with job applicants sent to them from different one-stop programs. Finally, state and local officials expressed concern that a large portion of one-stop participants are not included in performance measures. Many job seekers use self-service and informational activities but they are not tracked and counted in any program measures. While staff time and resources are used to establish and maintain self-service resource rooms and web sites, job seekers who use only these services will not be included in any of the performance measures. Without any information on individuals who use self-service, it will be difficult for Labor to show how effectively one-stops are being used. Multiple Measures Cannot Be Used to Measure the One-Stop System’s Performance: Performance measures for different programs often track similar outcomes, as figure 4 shows. However, the measures cannot be combined to obtain an overall view of one-stop performance. Although the same terms are used in various performance measures, their definitions are not identical. For example, while WIA older and younger youth programs define youth as being between the ages of 14 and 21, the laws governing Job Corps and HUD’s Youthbuild define youth as being between the ages of 16 and 24. Similarly, the definition of veterans is different for the Employment Service and Veterans’ Employment and Training Service program. The differences in definitions mean that assessing the outcomes for youth or veterans by combining the performance measures of individual programs within the one-stop setting would be difficult. Figure 4: Comparison of Performance Measures for 17 One-Stop Programs: [Refer to PDF for image] This figure illustrates the following information: Department of Labor: WIA Adult[C]; Placement in employment: [Check]; Retention in employment[A]: [Check]; Earnings: [Check]; Skill improvement[B]: [Check]; Customer satisfaction: [Check]; Other: [Empty]. Department of Labor: WIA Dislocated Worker[C]; Placement in employment: [Check]; Retention in employment[A]: [Check]; Earnings: [Check]; Skill improvement[B]: [Check]; Customer satisfaction: [Check]; Other: [Empty]. Department of Labor: WIA Youth (age 14-21)[C]; Placement in employment: [Check]; Retention in employment[A]: [Check]; Earnings: [Check]; Skill improvement[B]: [Check]; Customer satisfaction: [Check]; Other: [Empty]. Department of Labor: Wagner-Peyser Employment Service; Placement in employment: [Check]; Retention in employment[A]: [Check]; Earnings: [Empty]; Skill improvement[B]: [Empty]; Customer satisfaction: [Check]; Other: [Empty]. Department of Labor: Trade Adjustment Assistance Training Program[D]; Placement in employment: [Check]; Retention in employment[A]: [Check]; Earnings: [Check]; Skill improvement[B]: [Empty]; Customer satisfaction: [Empty]; Other: [Empty]. Department of Labor: Employment and training services to Veterans[E]; Placement in employment: [Check]; Retention in employment[A]: [Check]; Earnings: [Empty]; Skill improvement[B]: [Empty]; Customer satisfaction: [Empty]; Other: [Check]. Department of Labor: Unemployment Insurance[F]; Placement in employment: [Empty]; Retention in employment[A]: [Empty]; Earnings: [Empty]; Skill improvement[B]: [Empty]; Customer satisfaction: [Empty]; Other: [Empty]. Department of Labor: Job Corps; Placement in employment: [Check]; Retention in employment[A]: [Check]; Earnings: [Check]; Skill improvement[B]: [Check]; Customer satisfaction: [Empty]; Other: [Empty]. Department of Labor: Welfare-to-Work Program; Placement in employment: [Check]; Retention in employment[A]: [Check]; Earnings: [Check]; Skill improvement[B]: [Empty]; Customer satisfaction: [Empty]; Other: [Empty]. Department of Labor: Senior Community Service Employment Program (SCSEP)[G]; Placement in employment: [Check]; Retention in employment[A]: [Check]; Earnings: [Empty]; Skill improvement[B]: [Empty]; Customer satisfaction: [Check]; Other: [Check]. Department of Labor: Migrant and Seasonal Farmworker Employment and Training Program[H]; Placement in employment: [Check]; Retention in employment[A]: [Check]; Earnings: [Check]; Skill improvement[B]: [Check]; Customer satisfaction: [Check]; Other: [Empty]. Department of Labor: Native American Employment and Training Programs[I]; Placement in employment: [Empty]; Retention in employment[A]: [Empty]; Earnings: [Empty]; Skill improvement[B]: [Empty]; Customer satisfaction: [Empty]; Other: [Check]. Department of Education: Vocational Rehabilitation Services Program; Placement in employment: [Check]; Retention in employment[A]: [Empty]; Earnings: [Check]; Skill improvement[B]: [Empty]; Customer satisfaction: [Empty]; Other: [Check]. Department of Education: Adult Education and Literacy; Placement in employment: [Check]; Retention in employment[A]: [Check]; Earnings: [Empty]; Skill improvement[B]: [Check]; Customer satisfaction: [Empty]; Other: [Check]. Department of Education: Carl D. Perkins Vocational Education Program[J]; Placement in employment: [Check]; Retention in employment[A]: [Check]; Earnings: [Empty]; Skill improvement[B]: [Check]; Customer satisfaction: [Empty]; Other: [Empty]. HHS: Community Services Block Grant[K]; Placement in employment: [Check]; Retention in employment[A]: [Check]; Earnings: [Check]; Skill improvement[B]: [Empty]; Customer satisfaction: [Empty]; Other: [Check]. HUD: HUD-administered employment and training (Youthbuild); Placement in employment: [Empty]; Retention in employment[A]: [Empty]; Earnings: [Empty]; Skill improvement[B]: [Check]; Customer satisfaction: [Empty]; Other: [Check]. [A] May include retention in education and/or training. [B] May include the attainment of a credential or placement in education or training. [C] Two measures for customer satisfaction—one for employers, one for job seekers—are used for all WIA-funded programs. [D] This program reports by submitting individual records on program participants. [E] Based on proposed regulations. [F] Unemployment Insurance does not have participant performance measures. [G] SCSEP is required to develop performance measures, which are expected to be in place for program year 2003, according to a Labor official. [H] Grantees for this program are required to use the core indicators of performance common to WIA adult and youth programs. [I] Individual grantees propose performance in their individual grantee plans, which Labor must approve. [J] Measures must include either placement or retention and may include both. [K] For this program, states must report on at least one measure from each of six national goals. The six national goals are (1) number employed, (2) number maintaining 90-day retention, (3) number making progress toward literacy/GED, (4) number gaining health care coverage through employment, (5) increase in the availability and affordability of essential services, and (6) increase in access to community services and resources by low-income people. States may select from among national measures or develop their own. Source: Based on agency regulations and documents. [End of figure] Besides variations in definitions, there are also variations in how measures are calculated in different programs. For example, while the entered employment rate for WIA’s adult program is defined as the percentage of workers who get a job by the end of the first quarter after exit, the entered employment rate for the Employment Service is defined as the percentage of workers who get a job or changed employers in the first or second quarter after registration. As a result, performance data from these separate programs cannot be combined to yield a single overall score to assess various performance outcomes of the one-stop system. The Office of Management and Budget (OMB) convened a work group representing federal WIA partners to look at common definitions and measures across programs, which may issue guidance to states and localities on WIA performance measures and federal requirements. Separate Measures May Impede Coordination: Beyond failing to provide an overall picture of one-stop performance, state officials reported that separate performance measures impede the cooperation of one-stop partners.[Footnote 28] As a result, even though WIA was meant to establish a more coordinated workforce development system through the use of one-stops, over one-third of states surveyed expressed concern that individual program performance measures may impede this process. Some states even believed that separate measures caused competition among programs. They said that if staff did not understand that a participant could be counted in more than one program, they might not direct them to other one-stop programs. For example, one state reported in its written comments to our survey that competition for participants and duplication of services due to lack of coordination with other programs would continue as long as each program is required to meet its own performance and participation levels. Fourteen states volunteered in their written comments to our survey that the federal government should work to coordinate performance measures across programs or develop systemwide measures. In addition, while states agree that systemwide measures are needed, they caution against making any additional measures mandatory, since states are still adapting to the existing measures. Limited Progress on Developing Performance Measures for One-Stop Systems: Although WIA did not establish one-stop measures, it does require that Labor develop additional optional measures to assist states in assessing progress toward their workforce investment goals,[Footnote 29] which Labor has interpreted to include one-stop measures. Labor began developing workforce development performance measures to capture overall one-stop use after one-stop systems were piloted. Since the passage of WIA, Labor has continued its efforts to develop systemwide measures, but it has made limited progress. Labor convened a working group in September 2001 to develop additional indicators of one-stop performance. Partner representatives at this group included national workforce-related organizations including the National Association of State Workforce Agencies, National Governors’ Association, National Association of Counties, U.S. Conference of Mayors, and representatives of states and regional boards. This group is working to develop a menu of indicators that will help provide a comprehensive picture of WIA system activity. Such measures may include capturing information on self-service customers and the cost of services at the one-stop. [Footnote 30] These measures, while optional, would help provide information on overall one-stop use across the country if all states report on at least some of the measures. Labor plans to have guidelines for these optional indicators in place for use in program year 2002, which begins on July 1, 2002. In order for states to be able to implement them for the coming program year, Labor will need to provide final guidance well before July 2002. In addition to these national efforts, some states, on their own initiative, have attempted to develop additional measures for one-stop systems, but these efforts are not coordinated and do not allow for nationwide assessment of the one-stop system. According to our survey, eight states have created or are developing additional systemwide measures, but of those, only three are reporting them to Labor. Pennsylvania, for example, developed five measures specific to its one- stop system’s performance. These indicators, intended to measure the overall effectiveness of the one-stop system, include median cycle time to fill a job, and the percentage of employers and individuals using services through the one-stop. Florida, on the other hand, has developed “tiers” of measures that focus on the outcome of their workforce development programs. In the first tier, state-generated systemwide indicators measure many employment and training programs together. The second tier clusters similar types of programs and captures measures relevant to particular groups (e.g., continued education status of youth in youth programs). The third tier captures all the federally mandated measurements, as well as measures for the other tiers, such as caseloads for specific programs. In this way, Florida has attempted to measure the system overall as well as outcomes for individual programs. Measures developed by other states include the number of people using the resource rooms at one-stops and the increase over time in the number of unemployed people getting a job. Despite these states’ efforts, the absence of nationally established systemwide measures means that Labor cannot ensure nationwide comparability. Conclusions: WIA represents a fundamental shift in the way federally funded employment and training services are provided to job seekers and in the way WIA programs measure and monitor success. Despite obstacles, in just over a year states have made good progress in implementing the new requirements under WIA—developing new processes and designing new systems. Labor, for its part, has been working to find ways to allow states and localities greater flexibility to design their programs to meet local needs and has been actively seeking opportunities for states to have input into the process, particularly in the area of performance measurement. But given the challenges states have faced in implementing the new performance measurement system, more time is needed before the measures can meaningfully gauge the success of the programs. This new performance measurement system under WIA is a high-stakes game—a state’s future funding and, therefore, its ability to serve its citizens may depend upon how well it performs compared to how well it is expected to perform. It should be no surprise that states and localities are designing their systems and processes in ways that will enhance their ability to meet their performance levels. Because states see the current performance levels as too high for the current economy, states and localities may choose not to serve those job seekers who may be helped by their services, but who may not help in achieving their negotiated performance levels. Unless the performance levels can be adjusted to truly reflect differences in economic conditions and the population served, local areas will continue to have a disincentive to serve some job seekers that could be helped. WIA’s requirement to use UI data to track outcomes is a step in the right direction—it provides federal, state, and local government entities with an objective means to evaluate program success. But it brings challenges that need to be addressed, and states will need help to do so. Establishing the means to routinely share data across state lines through WRIS and developing ways to share promising approaches in the use of supplemental data sources and in managing the assessment of short-term program needs would go far in moderating these challenges. Without this help and the cooperative efforts of states and localities toward this end, developing a useful performance measurement system will take longer and cost more. In general, WIA’s performance measurement system captures some useful information, but it may not capture all the right information. The measure to track credentials has limited value because it lacks a standard definition for what’s being measured. For other measures, the lack of clear definitions for whom to track limits their usefulness in drawing conclusions about program success at both the state and national levels. Without clear definitions and processes, the measures will not provide the Congress with a true picture of how well the programs are performing. Furthermore, WIA performance measures gauge only WIA-funded services; yet there is widespread agreement that measures are needed to gauge the effectiveness of the entire one-stop system. The system’s narrow focus on program outcomes for a limited number of participants misses a key requirement of WIA to support the movement toward a coordinated system. In fact, the measures may foster the opposite—a siloed approach that encourages competition among programs and limits their cooperation. Without global one-stop measures, the Congress will not be able to assess how well states and localities are doing in meeting WIA’s requirement to coordinate services. The lack of such measures may, instead, send a signal to states that service coordination is a minor goal. Recommendations for Executive Action: To give states and local areas more time to implement WIA performance measures and establish baseline data needed to determine performance levels, we recommend that the Department of Labor: * delay the application of financial sanctions for at least 1 year or until it is judged that states have their data systems sufficiently in place to successfully track WIA outcomes. To eliminate possible disincentives to serve some job seekers and ensure that states and local areas will not be unduly penalized for economic downturns, we recommend that the Department of Labor: * expedite the release of guidance on revising negotiated performance levels and allow states to immediately begin the process of re- negotiation. To ensure uniformity in data collection and reporting so that performance results are more accurate and comparable across states, we recommend that the Department of Labor: * provide clearer guidance using objective criteria on who should or should not be registered as a WIA participant for tracking purposes and, once the guidance is released, work proactively with states to implement it, and: * issue guidance delineating a clear definition for what constitutes a credential, and, once the guidance is released, ensure that states use it to report on this indicator. To help states address the challenges of using UI data to measure outcomes, we recommend that the Department of Labor: * continue to fully fund the Wage Record Interchange System in order to facilitate the sharing of UI data across state lines; * develop ways for states to share promising approaches in the use of supplemental data sources in closing the data gaps for covered and uncovered employment in UI; and; * develop ways for states to share promising approaches that help states address the UI timeliness issue, providing methods to help states monitor and improve their programs in a timely manner. To help states measure one-stop performance, we recommend that the Department of Labor: * ensure that the development of optional one-stop system measures is completed in enough time for states to implement them at the beginning of program year 2002. Agency Comments: We provided a draft of this report to Labor for its review and comment. Labor’s comments are in appendix II. We incorporated comments and clarifications where appropriate. Labor generally agreed with our findings and recommendations, noting that they are consistent with information they have gathered from state and local partners. In its comments, Labor expressed concern that negotiated performance levels may be determining who receives WIA- funded services, indicating that it will work with states and local areas to address this issue. Labor also commented on our finding regarding the lack of clear guidance on certain policies, stressing the importance of state and local flexibility in determining specific policies and practices to fit local needs. While state and local flexibility is important, we continue to be concerned that the lack of a uniform understanding of when registration occurs and what constitutes a credential raises questions about both the accuracy and the comparability of states’ performance data. We are pleased to note that Labor is in the process of reviewing this issue. Finally, Labor cites its efforts to collaborate with states and local areas in developing a performance accountability system and increasing partnerships. We commend Labor for obtaining states’ input and participation in developing such a system. We are sending copies of this report to the Secretary of Labor, relevant congressional committees, and others who are interested. Copies will also be made available to others upon request. The report is also available on GAO’s home page at [hyperlink, http://www.gao.gov]. Please contact me on (202) 512-7215 if you or your staff have any questions about this report. Other major contributors to this report are listed in appendix III. Signed by: Sigurd R. Nilsen: Director, Education, Workforce, and Income Security Issues: [End of section] Appendix I: WIA Performance Measures for Adults, Dislocated Workers, and Older Youth: The following table includes descriptions of the performance measures reviewed in this report. It does not include the calculations required to attain the final value of the performance measures. Adults: Entered employment rate; Definition: Of those who did not have a job when they registered for WIA, the percentage of adults who got a job by the end of the 1st quarter after exit. This measure excludes participants who are employed at the time of registration. Adults: Employment retention rate at 6 months; Definition: Of those who had a job in the 1st quarter after exit, the percentage of adults who have a job in the 3rd quarter after exit. Adults: Average earnings change in 6 months; Definition: Of those who had a job in the 1st quarter after exit, the post-program earnings increases as compared with pre-program earnings. Adults: Employment and credential rate; Definition: Of those adults who received WIA training services, the percentage who were employed in the 1st quarter after exit and received a credential by the end of the 3rd quarter after exit. Dislocated workers: Entered employment rate; Definition: The percentage of dislocated workers who got a job by the end of the 1st quarter after exit. This measure includes dislocated workers who are employed at the time of registration. Dislocated workers: Employment retention rate at 6 months; Definition: Of those who had a job in the 1st quarter after exit, the percentage of dislocated workers who have a job in the 3rd quarter after exit. Dislocated workers: Earnings replacement rate in 6 months; Definition: Of those who had a job in the 1st quarter after exit, the percentage of pre-program earnings being earned post-program. Since it may be difficult to find dislocated workers jobs with equivalent or better wages, this measure captures the percentage of earnings of the new job in relation to the old. Dislocated workers: Employment and credential rate; Definition: Of those dislocated workers who received WIA training services, the percentage who were employed in the 1st quarter after exit and received a credential by the end of the 3rd quarter after exit. Older youth (age 19-21): Entered employment rate; Definition: Of those who are not employed at registration and who are not enrolled in post-secondary education or advanced training in the 1st quarter after exit, the percentage of older youth who have gotten a job by the end of the 1st quarter after exit. This measure also excludes youth that move on to post-secondary education or advanced training and not employment. Older youth (age 19-21): Employment retention rate at 6 months; Definition: Of those who are employed in the 1st quarter after exit and who are not enrolled in post-secondary education or advanced training in the 3rd quarter after exit, the percentage of older youth that are employed in the 3rd quarter after exit. Older youth (age 19-21): Average earnings change in 6 months; Definition: Of those who had a job in the 1st quarter after exit and who are not enrolled in post-secondary education or advanced training, the post-program earnings increases as compared with pre-program earnings. Older youth (age 19-21): Employment/education/training and credential rate; Definition: The percentage of older youth who are in employment, post- secondary education, or advanced training in the 1st quarter after exit and received a credential by the end of the 3rd quarter after exit. Customer satisfaction[A]: Employer customer satisfaction; Definition: The average of three statewide survey questions rated 1-10: (1 being “very dissatisfied” and 10 being “very satisfied”): * was the employer satisfied with services; * did the service meet the expectations of the customer; * how well did the service compare to the ideal set of services. Customer satisfaction[A]: Participant customer satisfaction; Definition: The average of three statewide survey questions rated 1-10: * was the participant satisfied with services; * did the service meet the expectations of the customer; * how well did the service compare to the ideal set of services. [A] A statewide telephone survey of a sample of 500 is conducted for all the WIA-funded programs. Source: U.S. Department of Labor TEGL 7-99. [End of table] [End of section] Appendix II: Comments From the Department of Labor: U.S. Department of Labor: "A Proud Member of America's Workforce Network" Assistant Secretary for Employment and Training: Washington, D.C. 20210: January 16, 2002: Mr. Sigurd R. Nilsen: Director: Education, Workforce and Income Security Issues: U.S. General Accounting Office: Washington, DC 20548: Dear Mr. Nilsen: Thank you for the opportunity to provide comments on the draft GAO report, Workforce Investment Act: Improvements Needed in Performance Measures to Provide More Accurate Picture of WIA's Effectiveness (GAO- 02-275). In general, GAO's findings and recommendations are consistent with what we have heard in discussions of the performance accountability system both within the Department and with our state and local partners. The Department of Labor's approach to performance accountability under title I of the Workforce Investment Act (WIA) has been based on: (1) the system change created by WIA; (2) state and local flexibility; and (3) increased partnerships. The System Change Created by WIA: The enactment of WIA brought dramatic changes to many aspects of the nation's workforce system. Many of these changes were in the area of service delivery. WIA streamlined eligibility and service strategy procedures in order to allow individuals to be served in an expedited and customer-focused manner. The delivery of services to customers is the first priority, and administrative processes such as reporting and paperwork become less important than program performance. The discussion on pages 14 and 15 of the draft related to states reporting that performance levels may determine who receives WIA-funded services causes us concern that not all areas are taking advantage of the opportunities that WIA provides. In 2002, ETA staff will work closely with states and local areas to clarify any misconceptions in this crucial area and to stress that the priority of the workforce investment system is to provide quality services to customers. With the focus on service delivery, performance accountability becomes even more important. For example, WIA contains sets of core indicators designed to measure the youth, adult, and dislocated worker programs, as well as customer satisfaction indicators. While we agree that measures which assess the performance of the One-Stop system as a whole are needed, WIA does not contain comprehensive measures for the One- Stop delivery system. As noted in your report, last September we began to gather information and recommendations on system measures to support state use of additional performance indicators in preparation for WIA reauthorization discussions. State and Local Flexibility: In implementing WIA, the Department has developed policies and guidance designed to support the key WIA reform principle of increased state and local flexibility. Accordingly, when developing the regulations and performance accountability guidelines, in many instances the Department set broad parameters or definitions, leaving the process of establishing specific policies or definitions up to the states and local areas. Two examples of this flexibility are touched upon in the draft report: (1) the point of registration for adults and dislocated workers, and (2) the definition of a credential for performance accountability purposes. As you noted in your report, we have provided guidance on establishing a point of registration and on the states' authority to define credentials, but in both cases the needs of the job seeker and employer customers at the state and local level were given priority over a one-size-fits-all national requirement. Based on state and local input, the Department is in the process of reviewing these policy decisions. No decision regarding revisions to these existing policies has been made at this time. Increased Partnership: WIA also calls for increased partnerships between all levels of the workforce system. Accordingly, the WIA title I performance accountability system has been developed through an unprecedented collaborative process. The Department has made great efforts to ensure that the performance accountability system for title I was designed in partnership with states and local areas. A series of consultation papers were published for public comment in the Federal Register beginning in March 1999. The March 1999 consultation paper specifically asked for comment on several of the issues discussed in the draft report, such as (1) the point at which adult and youth registrants are counted for different performance measures; (2) the use of wage records for performance measurement considering availability, completeness, and accuracy; and (3) identifying possible unintended effects resulting from definitions/policies around performance measurements. The public was also provided an opportunity to comment on the interim final regulations, published in April 1999, which covered the point of registration issue and the performance accountability system. The Department hosted two rounds of town hall meetings to solicit comments from the public on all issues related to the implementation of WIA, including the area of performance. Proposed WIA title I reporting requirements were published in the Federal Register for comment, and consultation with workforce partners continued to ensure that all comments and points of view had been considered. We are impressed by the progress that our state and local partners have made in implementing the performance accountability provisions of WIA title I. However, due to the complexity of building a comprehensive and seamless workforce system, we anticipate that the performance infrastructure will continue to develop over the next several years, and that lessons learned from states and local areas will be used to improve the system as it evolves. We have enclosed some additional specific comments on the GAO's findings. We will provide a formal response to GAO's recommendations for executive action when the report is final. In addition, we expect that the issue of performance accountability will be a key issue analyzed as part of the upcoming WIA reauthorization discussions. Sincerely, Signed by: Emily Stover DeRocco: Enclosure: [End of section] Appendix III: GAO Contacts and Staff Acknowledgments: GAO Contacts: Dianne Blank (202) 512-5654: Ronni Schwartz (202) 512-7033: Staff Acknowledgments: Abbey Frank, Mikki Holmes, and Amanda Ahlstrand made significant contributions to this report. In addition, James Wright assisted in the study design and the national survey; Jessica Botsford and Richard Burkard provided legal support; and Patrick DiBattista and Barbara Alsip assisted in the message and report development. [End of section] Related GAO Products: U.S. General Accounting Office. Workforce Investment Act: Better Guidance Needed to Address Concerns Over New Requirements. [hyperlink, http://www.gao.gov/products/GAO-02-72]. Washington, D.C.: 2001. U.S. General Accounting Office. Veterans’ Employment and Training Service: Proposed Performance Measurement System Improved, But Further Changes Needed. [hyperlink, http://www.gao.gov/products/GAO-01-580]. Washington, D.C.: 2001. U.S. General Accounting Office. Multiple Employment and Training Programs: Overlapping Programs Indicate Need for Closer Examination of Structure. [hyperlink, http://www.gao.gov/products/GAO-01-71]. Washington, D.C.: 2000. U.S. General Accounting Office. Workforce Investment Act: Implementation Status and the Integration of TANF Services. [hyperlink, http://www.gao.gov/products/GAO/T-HEHS-00-145]. Washington, D.C.: 2000. [End of section] Footnotes: [1] In addition to the Department of Labor, which oversees the implementation of WIA, programs funded through the Departments of Education, Health and Human Services, and Housing and Urban Development are required to provide services for certain of their programs through this one-stop center system. [2] While WIA reauthorizes or amends other existing programs, including Adult Education and Literacy, Employment Service (Wagner-Peyser), and Vocational Rehabilitation, the performance measures apply only to the adult, dislocated worker, and youth programs under Title I, which, for this report, we will refer to as “WIA-funded programs.” [3] States must also meet performance levels for programs authorized under the Adult Education and Family Literacy Act and Carl D. Perkins Vocational and Applied Technology Education Act (Perkins) to be eligible for incentive grants. [4] We also did not review measures for eligible training providers, which were reviewed in our study, U.S. General Accounting Office, Workforce Investment Act: Better Guidance Needed to Address Concerns Over New Requirements, [hyperlink, http://www.gao.gov/products/GAO-02-72] (Washington, D.C.: 2001). [5] WIA’s Youth program uses low income as an eligibility requirement. [6] Authorized through fiscal year 2003, WIA’s three new programs had a budget authority of about $3.7 billion in fiscal year 2001. [7] GPRA was intended to focus government decision making, management, and accountability on the results and outcomes achieved by federal programs. [8] Guidance from Labor defines a credential as a nationally recognized degree or certificate or a recognized state/locally defined credential. [9] According to Labor’s guidance, states must achieve a cumulative score of 100 percent or more in each program area—adult, dislocated worker, and youth—and for customer satisfaction and at least 80 percent of the negotiated performance level for all 17 measures. Failing to achieve the 80-percent level for any one performance measure disqualifies a state from being considered for an incentive grant. [10] According to Labor, while the agency has sole responsibility for administering sanctions under WIA, awarding incentives is a joint effort involving Labor and the Department of Education. [11] In addition, three other categories of programs are required to provide services through the one-stop center: Youth Opportunity Grants; demonstration, pilot, multi-service, research, and multi-state projects; and national emergency grants. Because they are of limited scope, we did not include them in our total. [12] WIA requires states to establish and operate management information systems based on guidelines established by Labor, which are designed to promote the efficient collection of information. [13] U.S. Department of Labor Employment and Training Administration, Training and Employment Guidance Letter No. 14-00 (Mar. 5, 2001). [14] Calculations for the measures are found in U.S. Department of Labor Employment and Training Administration, Training and Employment Guidance Letter (TEGL) No. 7-99 (Mar. 3, 2000). [15] Dislocated workers may include those who have obtained interim employment after being displaced. [16] States must wait two full quarters after exit before collecting data because of delays in available wage records. [17] Data for pre-program earnings may be collected at registration or at any time until the fifth quarter after exit. [18] As authorized by WIA, Labor requires states to submit quarterly summary reports reflecting the state’s performance levels and other activities. [19] The 23 states are a subset of the 53 states and territories that report WIA data to Labor. [20] While JTPA focused on providing training to the economically disadvantaged, including the hardest-to-serve, WIA provides a broader range of services to all individuals, regardless of their eligibility for other services. [21] The adjusted levels would apply only to the dislocated workers from the company that has been affected, not to the entire local area. [22] TEGL No. 7-99. [23] All youth who receive WIA-funded services are required to be registered. [24] See U.S. General Accounting Office, Veterans’ Employment and Training Service: Proposed Performance Measurement System Improved, But Further Changes Needed, GAO-01-580 (Washington, D.C.: 2001). [25] To participate, states are required to be technically and administratively ready. For example, states must be able to respond to queries for UI wage data and establish internal security measures to ensure the confidentiality of data. [26] A national association is currently working a contractor to study the costs associated with WRIS. [27] Labor’s Office of Youth Services has taken action to help youth programs in this area, such as creating a set of short-term performance measures. [28] This issue was also raised in a study that found that the continued emphasis on measuring performance in categorical programs undermines the integration expected in WIA. Evelyn Ganzglass, Martin Jensen, Neil Ridley, Martin Simon, and Chris Thompson. Transforming State Workforce Development Systems: Case Studies of Five Leading States (National Governors’ Association, 2001). [29] Section 136 (i)(1) requires Labor to work with workforce development system partners to develop terms for a menu of additional indicators of performance to help states assess their progress toward their workforce investment goals. The purpose of developing these additional measures is to ensure nationwide comparability. 30Labor is currently seeking OMB approval to collect data on the costs and usage of WIA and Wagner-Peyser funded services that do not require registration. Labor issued a Federal Register Notice on January 16, 2002 seeking public comment on this proposal. [End of section] GAO’s Mission: The General Accounting Office, the investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through the Internet. GAO’s Web site [hyperlink, http://www.gao.gov] contains abstracts and full text files of current reports and testimony and an expanding archive of older products. The Web site features a search engine to help you locate documents using key words and phrases. You can print these documents in their entirety, including charts and other graphics. Each day, GAO issues a list of newly released reports, testimony, and correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site daily. The list contains links to the full-text document files. To have GAO e-mail this list to you every afternoon, go to [hyperlink, http://www.gao.gov] and select “Subscribe to daily E-mail alert for newly released products” under the GAO Reports heading. Order by Mail or Phone: The first copy of each printed report is free. Additional copies are $2 each. A check or money order should be made out to the Superintendent of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a single address are discounted 25 percent. Orders should be sent to: U.S. General Accounting Office: 441 G Street NW, Room LM: Washington, D.C. 20548: To order by Phone: Voice: (202) 512-6000: TDD: (202) 512-2537: Fax: (202) 512-6061: To Report Fraud, Waste, and Abuse in Federal Programs Contact: Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: E-mail: fraudnet@gao.gov: Automated answering system: (800) 424-5454 or (202) 512-7470: Public Affairs: Jeff Nelligan, managing director, NelliganJ@gao.gov: (202) 512-4800: U.S. General Accounting Office: 441 G Street NW, Room 7149: Washington, D.C. 20548: