This is the accessible text file for GAO report number GAO-04-308 entitled 'Workforce Investment Act: Labor Actions Can Help States Improve Quality of Performance Outcome Data and Delivery of Youth Services' which was released on February 23, 2004. This text file was formatted by the U.S. General Accounting Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. Report to the Ranking Minority Member, Committee on Health, Education, Labor, and Pensions, U.S. Senate: United States General Accounting Office: GAO: February 2004: WORKFORCE INVESTMENT ACT: Labor Actions Can Help States Improve Quality of Performance Outcome Data and Delivery of Youth Services: GAO-04-308: GAO Highlights: Highlights of GAO-04-308, a report to the Ranking Minority Member, Committee on Health, Education, Labor, and Pensions, U.S. Senate Why GAO Did This Study: The Workforce Investment Act (WIA) has been in effect for several years and is currently undergoing reauthorization. In order to provide the Congress with information on the implementation and effectiveness of the WIA youth program, GAO was asked to explore how services have been delivered at the local level, whether the Department of Labor’s guidance has addressed challenges faced by local areas, and how effective the program has been. What GAO Found: Local areas primarily used the WIA program for dropout prevention and other efforts to improve academic achievement for in-school youth. Nationally, about 70 percent of youth served were in school, but percentages ranged from 38 to 86 percent by state. Officials in the five states GAO visited said that they focused on in-school youth because serving out-of-school youth was much more difficult and expensive, and less effective. Local areas emphasized learning-related summer employment for in-school youth and occupational skills training and supportive services for out-of-school youth. Over half of local boards nationwide used providers that had subcontracting arrangements with others to deliver youth services. The majority of youth were served primarily from educational institutions and community organizations. Despite Labor’s guidance, local areas continue to face implementation challenges in identifying and retaining out-of-school youth, providing youth with mentoring and follow-up services, and using interim measures for ongoing program assessment. While Labor supports information exchange forums, a promising practices Web site, and technical assistance, some local areas may have difficulties gaining access to and using these resources. Little is known about the effectiveness of the WIA youth program because Labor has not yet conducted an impact evaluation. In addition, while the youth program exceeded most of its performance goals, these data were questionable because of problems with state information systems and inadequate oversight of data quality. While states will be required to verify data, concerns remain about their ability to fully implement the requirement and Labor’s ability to monitor implementation consistently. What GAO Recommends: GAO is recommending that the Departments of Labor and Education coordinate efforts to clarify how schools can work with workforce officials to help connect school dropouts to local WIA youth programs. GAO is also recommending that the Department of Labor provide states and local areas with technical assistance necessary to address ongoing implementation challenges and establish standard monitoring procedures to improve the quality of data reported by states. In formal comments on a draft of this report, Education concurred with our recommendation to work with Labor to connect out-of-school youth to local WIA youth programs. In its informal comments, Labor said that many of the findings corroborated its own observations and that the recommendations are consistent with Labor’s current program direction. www.gao.gov/cgi-bin/getrpt?GAO-04-308. To view the full product, including the scope and methodology, click on the link above. For more information, contact David Bellis (415) 904-2272 or bellisd@gao.gov. [End of section] Contents: Letter: Results in Brief: Background: WIA Program Used Primarily for Dropout Prevention and Emphasized Summer Services and Skills Training Coordinated among Multiple Service Providers: Labor's Guidance Has Not Fully Addressed Implementation Challenges Faced by Local Areas: Lack of Program Evaluations and Questionable Performance Data Prevent Assessments of WIA Youth Program: Conclusions: Recommendations for Executive Action: Agency Comments and Our Evaluation: Appendix I: Comments from the Department of Education: Appendix II: GAO Contacts and Staff Acknowledgments: GAO Contacts: Staff Acknowledgments: Related GAO Products: Tables: Table 1: WIA Youth Performance Measures: Table 2: Percentage of Local Boards Using Different Types of Service Delivery Arrangements, by Number of Youth Served: Figures: Figure 1: Percentage of All WIA Youth Served Who Were in School, by State (including the District of Columbia): Figure 2: WIA Services Provided to In-School and Out-of-School Youth: Figure 3: Number of WIA Youth Service Providers Used by Local Boards: Figure 4: Primary Providers of Services to WIA Youth: Abbreviations: OIG: Office of Inspector General: JTPA: Job Training Partnership Act: WIA: Workforce Investment Act: WIASRD: Workforce Investment Act Standardized Record Data: United States General Accounting Office: Washington, DC 20548: February 23, 2004: The Honorable Edward M. Kennedy: Ranking Minority Member: Committee on Health, Education, Labor, and Pensions: United States Senate: More than 5 million young people--15 percent of the nation's youth population between ages 16 and 24--are out of school and out of work, according to a recent study. Further, many teenagers and young adults who are in school are at risk of dropping out. According to some experts, this indicates that a considerable portion of the country's emerging workforce may face significant difficulty making the transition to productive adulthood.[Footnote 1] At the same time, the Department of Labor projects that some labor demands will go unmet because there will be too few workers in the labor market with the necessary skills. Enacted in 1998, the Workforce Investment Act (WIA) aims to address these issues by assisting the nation's emerging workforce in realizing its full potential. As the administering agency, Labor has budgeted about $1 billion annually on WIA Title I-B youth employment and training programs to serve an estimated 721,000 of the nation's most at-risk young people. WIA services are based on promising practices in the fields of youth development and employment. Research suggests, for example, that youth can achieve positive outcomes when they interact with caring adults, engage in hands-on education and training activities, and receive support for personal growth. The WIA program has sought to make these sorts of experiences available to both in-school and out-of-school youth participants. Under WIA, local areas can tailor their approach to the types of youth served, the services provided, and how they are delivered. To ensure that youth programs are tailored to local areas, WIA requires the participation of a wide variety of people--youth policy experts, representatives from youth-serving agencies, parents, and others with a vested interest in the local youth programs. These participants serve on local workforce investment boards created by WIA to establish workforce development policies and oversee implementation at the local level, or on youth councils, subgroups of the local board that plan and coordinate the local youth program. We previously reported on the implementation challenges local areas faced during the first few years of implementation.[Footnote 2] Now that the program has been ongoing for several years and WIA is undergoing reauthorization, you asked us to review (1) what approaches local areas have taken to serve at-risk youth, (2) whether Labor's youth program guidance has addressed ongoing implementation challenges, and (3) what is known about the effectiveness of the WIA youth program. To obtain information on what approaches local areas have taken to deliver youth services, we administered a survey to the directors of all 604 local workforce investment boards across the nation, including those in the 50 states, the District of Columbia, Puerto Rico, and U.S. territories. We received responses from 496 local workforce investment boards (82 percent) and relied on self-reported data. To further understand local area approaches to service delivery, we visited nine local workforce investment areas in five states: California, Louisiana, New Hampshire, Ohio, and Virginia. We selected these states to obtain a mix based on their differences in geographic location, number of workforce investment areas, amount of youth funding, and presence of a state youth council. In addition, our selection was informed by recommendations from Labor, youth policy experts, and state workforce officials. We interviewed officials representing state and local boards, youth councils, one-stop centers, youth-service providers, business representatives, and state and local education agencies. We reviewed Labor's program evaluation agenda and published guidance letters from program year 2000 to the present. We also assessed the reliability of the performance data submitted by states in their annual reports and compiled in Labor's WIA database known as the Workforce Investment Act Standardized Record Data (WIASRD) by performing checks for internal consistency, reviewing Office of Inspector General reports and other relevant documents, and speaking with the contractor for WIASRD. We determined that the data were not sufficiently reliable to use for the purposes of this report. We also interviewed officials from Labor, including each of its six regional offices, as well as from the Department of Education and national associations. Our work was conducted between January and December 2003 in accordance with generally accepted government auditing standards. Results in Brief: Local WIA youth programs primarily focused on preventive strategies to help in-school youth avoid academic failure and dropping out of school, emphasizing summer enrichment services that were coordinated through individual case managers and multiple service providers. Our survey of local boards showed that about 70 percent of WIA youth were in school, although this percentage varied widely, from 38 percent in South Dakota to 86 percent in Nebraska. State and local education officials said that the WIA program allowed communities to provide complementary services needed to support at-risk in-school youth. For example, local officials from one urban area in Virginia said that while a local school provided 1 counselor for as many as 300 students, the WIA program funded 1 counselor for every 50 WIA participants. In addition, local officials in four states we visited said they focused on serving in-school youth because it was easier and less costly than recruiting and retaining youth once they had dropped out of school. Local boards reported that more than half of in-school youth received summer employment services that were linked to classroom learning. For example, a service provider in a rural area of California enrolled in- school youth in a 6-week summer enrichment program where students worked part-time while learning reading skills. In contrast, WIA youth who were out of school were more likely to receive occupational skills training and supportive services, such as assistance with child care, transportation, and housing. While in-school and out-of-school youth usually participated in programs separately, both groups worked with case managers who helped develop individual service strategies and coordinate delivery of services. The majority of WIA youth were served primarily by community organizations or by educational institutions such as high schools, colleges, and universities. Despite guidance issued by Labor, local areas continued to face challenges in serving out-of-school youth, providing mentoring and follow-up services, and establishing and using optional interim performance measures. One reason serving out-of-school youth continued to be challenging was that such youth can be difficult to locate within the community. Some local workforce and regional Labor officials said that identifying youth was problematic, in part because schools did not always ensure that dropouts were linked with the WIA program. Another reason cited was the difficulty in retaining youth who were primarily interested in immediate employment rather than in participating in WIA's long-term youth development activities. Local officials added that mentoring and follow-up services were challenging to provide, in part because of the difficulty in finding enough qualified mentors to work with at-risk youth and sustaining a connection with youth once they had exited the program. Some state workforce officials said that to more effectively serve out-of-school youth, they needed better- targeted guidance from Labor that addressed their local areas' particular service delivery issues. Local areas also faced challenges establishing interim measures, such as enrollment and service participation, to assess program performance and improve service delivery in a timely manner. Despite Labor's encouragement, some regional Labor officials said that interim performance measures were not widely used by local areas. Labor has established WIA youth learning exchanges, a Web site, and a technical assistance program to help address WIA youth program implementation challenges, but some states and local areas may not have access to or be aware of these resources. Little is known about the effectiveness of the WIA youth program because program impact evaluations have not been performed and performance outcome data that have been collected may not be reliable. Impact evaluations provide information on program effectiveness by differentiating between outcomes that result from the program itself and those that result from other factors. Because of possible legislative changes to the WIA youth program, Labor does not plan to undertake an impact evaluation until 2006, with initial results expected by 2009. While states annually report performance data to Labor, these data are questionable because of insufficient state monitoring of data quality and the inadequacies of some state management information systems. Labor officials said that they will require states to verify local area data beginning with program year 2002 data, but concerns remain about states' ability to fully implement validation requirements and Labor's ability to monitor implementation consistently. We are recommending that Labor and Education coordinate efforts to clarify how schools can work with workforce officials to connect dropouts to WIA youth services. We are also recommending that Labor provide guidance to address specific concerns identified by local implementers and establish standards to monitor data quality. Background: Enacted in 1998, WIA replaced the fragmented and overlapping programs under the Job Training Partnership Act (JTPA) with a system that sought to connect employment, education, and training services to better match job seekers to labor market needs. The youth program falls under Title I of WIA along with programs for adults and dislocated workers; Title II deals with adult and family literacy; Title III pertains to employment services under the Wagner-Peyser Act; and Title IV addresses vocational rehabilitation. To better prepare low-income youth who face barriers to employment and education, WIA requires youth programs to focus on long-term, comprehensive services delivered year-round through a coordinated network of service providers. In addition to meeting the low-income requirement, to be eligible for WIA services youth must be between the ages of 14 and 21 and faced with one or more of six barriers to employment.[Footnote 3] WIA serves both in-school and out- of-school youth. A significant portion of out-of-school youth are high school dropouts and unemployed youth who are struggling to succeed in the public education system and lack financial, family, or social support. Thus, at least 30 percent of local WIA youth funds must be spent on out-of-school youth.[Footnote 4] WIA Youth Funding: Since WIA's full implementation in 2000, funding for the youth program has ranged between approximately $1 billion and $1.4 billion a year. Labor follows a formula to allocate WIA funds to states, which in turn distribute money to their local workforce investment areas.[Footnote 5] At the state level, the governor can reserve up to 15 percent of the annual WIA allotment for such statewide activities as developing the WIA strategic plan for the state. States allocate the remaining 85 percent of funds to local boards that use the funds to develop the local strategic plan, establish a youth council, and award competitive contracts to youth service providers that have been recommended by the youth council. In addition to recommending eligible providers, the youth council coordinates youth activities in the local area, oversees providers, and carries out other duties authorized by the local board such as forging linkages with educational agencies. Youth councils may also leverage additional public and private funds to supplement their WIA funding in order to provide comprehensive youth services. Youth Services: Once they are determined to be WIA eligible, youth receive an objective assessment of their academic level, skills, and service needs. Local youth programs then use the assessment to create each participant's individual service strategy, which lays out employment goals, educational objectives, and necessary services. Every local youth program must offer the following 10 services, known as program elements, to eligible youth, though participants may receive different combinations of these elements depending on their service strategy. Labor groups the 10 required program elements around four major themes: Improving educational achievement: 1. tutoring, study skills training, and instruction leading to completion of secondary school, including dropout prevention, 2. alternative school services, Preparing for and succeeding in employment: 3. summer employment linked to academic and occupational learning, 4. paid and unpaid work experiences, including internships and job shadowing, 5. occupational skills training, Developing the potential of young people as citizens and leaders: 6. leadership development, which may include community service and peer-centered activities encouraging responsibility, Supporting youth: 7. supportive services (such as child care and housing assistance), 8. adult mentoring for at least 12 months that may occur both during and after program participation, 9. follow-up upon program completion for at least 12 months, and: 10. comprehensive guidance and counseling. Eligible youth may obtain these services directly from approved youth service providers or through WIA's one-stop system, which serves as a gateway to a variety of employment and training services.[Footnote 6] In addition to helping WIA youth gain access to the 10 program elements, the one-stop system also provides all youth with basic services, whether or not they are eligible for WIA. Any young person may walk into a one-stop center and make use of the center's career resources and obtain information on and referrals to other providers. While one-stop centers are designed primarily to serve those 18 and older, some states have established one-stops that serve only youth, as we reported previously.[Footnote 7] Youth Program Guidance: Labor provides guidance to help states and local areas implement WIA. Labor's guidance includes issuing annual Training and Employment Guidance Letters on the youth program in general as well as on specific topics, sponsoring WIA Learning Exchanges[Footnote 8] in every region, and maintaining a Web site for promising and effective practices. Labor responds to requests for clarification or additional information by phone or e-mail or at conferences. Labor's guidance often includes promising and effective practices to reinforce state and local flexibility under WIA to tailor programs to best meet youths' needs. Performance Measures: The law requires that states and local areas collect performance information on seven youth measures, which are separated for younger and older youth (see table 1). All seven youth measures apply to both statewide and local performance. Table 1: WIA Youth Performance Measures: Younger youth measures (ages 14-18): 1. Skill attainment rate; 2. Diploma or equivalent attainment rate; 3. Placement and retention in postsecondary education, advanced training, or employment; Older youth measures (ages 19-21): 1. Entered into employment rate; 2. The employment retention rate at six months; 3. Increase in average earnings; 4. The credential rate. Source: GAO analysis of the Workforce Investment Act. [End of table] WIA performance information is collected from service providers and flows upward to the local boards, then to state boards or agencies, and finally to Labor. Local performance data are aggregated and entered into a state's own automated data system that tracks the activities of individual WIA participants. States use two mechanisms to report performance to Labor: WIASRD, a database of individual records containing activity and outcome information for each registered participant that has exited WIA, and state annual reports. Labor uses the state annual reports to track states' progress in meeting negotiated performance goals. Labor then awards monetary incentives to states that meet or exceed their performance goals and sanctions states that fail to meet at least 80 percent of each goal in two consecutive years. WIA Program Used Primarily for Dropout Prevention and Emphasized Summer Services and Skills Training Coordinated among Multiple Service Providers: Most local areas used WIA youth program services for a range of approaches to prevent academic failure and school dropouts, emphasizing learning-related summer services that were coordinated through case managers and multiple service providers. Nationally, about 70 percent of youth served were in school, according to local boards responding to our survey. For in-school youth, local boards provided summer employment services linked to classroom learning more often than for out-of-school youth, while services for this latter population more often included occupational skills training and supportive services. Local boards reported that most youth received services primarily from community organizations and educational institutions. These providers were most likely to subcontract or make informal arrangements with other organizations to deliver the full range of WIA services to youth participants. Local Areas Focused on Dropout Prevention for In-School Youth: Overall, local areas' approach to serving youth was to supplement schools' dropout prevention efforts to keep youth connected to an educational system, according to state and local workforce and education officials. WIA allows states and local areas to determine the proportion of in-school and out-of school youth to be served, requiring only that they spend at least 30 percent of funds for out-of-school youth. Labor reported that with some exceptions, states chose to focus the majority of resources on in-school youth,[Footnote 9] and our survey showed that about 81 percent of local areas served more in- school youth than out-of-school youth. Nationally, about 70 percent of youth served were in school, according to local boards we surveyed. However, this percentage varied across the nation, as shown in figure 1. For example, 38 percent of youth served in South Dakota were in school, compared with 86 percent in Nebraska. Figure 1: Percentage of All WIA Youth Served Who Were in School, by State (including the District of Columbia): [See PDF for image] [A] Includes states where less than 65 percent of boards responded. [End of figure] According to officials in four states that we visited, WIA youth programs primarily targeted in-school youth because recruiting and retaining out-of-school youth for the WIA program was much more difficult and expensive. For example, officials from a rural area in Ohio had difficulty identifying and retaining out-of-school youth whose contact information changes frequently, and Labor reports that the average cost of serving an out-of-school youth under the WIA program is about $4,000 a year, twice as much as for an in-school youth. Officials from other local areas we visited considered other factors. A local official in Louisiana said that in-school youth are interested in the WIA program, unlike out-of-school youth, who are difficult to engage. In contrast, a rural area in Virginia chose to focus primarily on out- of-school youth because so few services were available for this population. The Administration has proposed amending the WIA youth program to focus more resources on out-of-school youth. Department of Education officials said that WIA's services to in-school youth were not unique, since schools already offer various services to their students to deter them from dropping out. While Education officials said that the department's grants geared exclusively toward dropout prevention would be insufficient to address the national dropout problem, they also stated that funding is available under many Education programs that could be used for dropout prevention activities.[Footnote 10] However, local workforce and education officials we spoke with in three states we visited said that they were either unaware of or unable to gain access to other available federal resources that could be used to provide intensive services to at-risk youth. In New Hampshire, for example, officials told us that WIA provides the only dropout prevention program, and that they were unaware of other available education dropout prevention resources. State and local workforce and education officials we spoke with in the five states we visited said that WIA funding complemented rather than duplicated education services and was critical in preventing in-school youth from dropping out of school. For example, in a rural area in Ohio, workforce officials stated that the WIA program was the only dropout prevention program and that WIA provided students with their only chance at academic and career success. In addition, they said that WIA's services were more intensive and comprehensive, and were delivered in a one-on-one setting where each student received individualized attention. In one urban school in Virginia, there was 1 WIA counselor for 50 eligible in-school youth, compared to 1 school counselor for as many as 300 students. In-School Youth Received More Summer Employment; Out-of-School Youth Received More Occupational Skills Training: Local areas tailored services provided to youth based on their needs, emphasizing different services, depending on whether youth were in or out of school. To meet youth's many developmental needs, the WIA youth program requires that local areas offer the same menu of 10 academic, employment, and support services to all eligible youth, which WIA providers choose from when tailoring services to an individual's service strategy based on an assessment of needs.[Footnote 11] As shown in figure 2, services such as work experience and leadership development were provided fairly equally to both youth populations, but there were differences in other areas. For example, in-school youth were more likely to receive tutoring services and summer employment linked to classroom learning, while out-of-school youth were more likely to receive occupational skills training and supportive services to help prepare them for employment. Figure 2: WIA Services Provided to In-School and Out-of-School Youth: [See PDF for image] Note: Follow-up services are not depicted in the figure because they are mandatory for all participants when they exit the WIA youth program. [End of figure] WIA requires that summer employment programs be linked to academic learning, and Labor guidance promotes meaningful summer enrichment experiences to help youth attain skills throughout the school year.[Footnote 12] Local youth programs in the five states we visited were providing youth with various types of summer experiences. For example, a service provider in a rural area of California enrolled in- school youth in a 6-week summer enrichment program where students worked part-time while learning reading skills. Another service provider in a rural area in Louisiana offered in-school youth summer services that included academic enrichment, community service, and exposure to different career options. Local youth programs provided out-of-school youth training in occupational skills and job readiness, as well as offered them supportive services. Labor's guidance states that enrolling youth in occupational skills training and retaining them until program completion leads to better outcomes.[Footnote 13] Local areas provided several different types of occupational skills training, depending on local labor market needs. For example, one urban local area in Ohio trained out-of-school youth in nursing, welding, and computer repair, while another urban local area in Virginia offered technology certification training. Local areas also provided job readiness skills cited by employers as lacking in many youth seeking employment. For example, a program for out-of-school youth in an urban area of Ohio taught participants such job readiness skills as customer service, conflict management, and other interpersonal skills. Supportive services were also important for out-of-school youth who needed additional assistance to help them overcome their multiple barriers to employment. A service provider in New Hampshire provided out-of-school participants with support services such as child care, transportation, and housing assistance. Case Managers Developed Service Strategies and Providers Coordinated to Deliver Services: Almost all local boards we surveyed reported using case managers to coordinate services for youth in and out of school. Because at-risk youth often require services from a host of departments, including Education, Health and Human Services, Housing and Urban Development, Justice, and others, Labor's guidance promotes the use of a case manager to coordinate services among all these youth-serving partners to provide seamless access to and delivery of services.[Footnote 14] Nationwide, all but 3 percent of local boards responding to our survey reported that their youth program uses case managers. Local boards reported that, in addition to performing other duties, case managers assessed youth upon enrollment in the WIA program. These assessments typically included a review of educational attainment, work readiness, work experience, and career interests. The assessments are then used to develop an individual service strategy for each youth participant. Labor's guidance requires that each strategy identify employment goals and educational objectives, and prescribe appropriate services for each participant.[Footnote 15] About three-fourths of local boards reported that youth were greatly or very greatly involved in the development of their individual service strategies. Most local areas used multiple service providers to deliver the youth services spelled out in each participant's individual service strategy. WIA does not specify how services must be provided to youth, allowing local areas to determine how many providers they will fund and hold responsible for delivering the services outlined in each youth's individual service strategy. Over half of local boards responding to our survey reported using 4 or fewer service providers, but 15 percent used more than 10 providers, as shown in figure 3. Figure 3: Number of WIA Youth Service Providers Used by Local Boards: [See PDF for image] Note: Some local board directors reported using zero providers because the board itself delivered youth services. WIA permits local boards to provide training services if there is an insufficient number of eligible providers to meet local demand and if the board has demonstrated that it meets the requirements for an eligible service provider. [End of figure] Over half of local boards nationwide used providers that did not deliver all services themselves, using formal or informal subcontracting arrangements to provide the range of services needed. The extent that providers coordinated with others to deliver services was related to the number of youth served in the local area. In local areas with 250 or fewer youth, providers delivered all services themselves more often than in areas with more than 250 youth, as indicated in table 2. Table 2: Percentage of Local Boards Using Different Types of Service Delivery Arrangements, by Number of Youth Served: Providers deliver all services without using subcontractors; 0-250 youth: 44; 251-750 youth: 34; Over 750 youth: 30. Providers are responsible for delivering all services but use subcontractors; 0-250 youth: 23; 251-750 youth: 31; Over 750 youth: 39. Providers do not deliver all services themselves, but make noncontractual arrangements with other organizations to make sure all services are delivered; 0-250 youth: 22; 251-750 youth: 25; Over 750 youth: 17. Other arrangement; 0-250 youth: 10; 251-750 youth: 10; Over 750 youth: 14. Source: GAO survey of local board directors. Note: Columns may not add to 100 because of rounding. [End of table] Regardless of the level of coordination used to provide youth services, workforce officials told us that providers often informally collaborated with one another through regular meetings to discuss problems, train one another, share best practices, or share other resources. Local boards responding to our survey reported using a range of public and private entities to deliver youth services. Over half of all youth received services primarily from educational institutions or community organizations, while less than 1 percent of youth received services primarily from faith-based organizations, as shown in figure 4. Figure 4: Primary Providers of Services to WIA Youth: [See PDF for image] [A] Youth opportunity centers were established under the Youth Opportunity Grant program to enable youth living in high-poverty areas to gain access to a wide range of services. [End of figure] Local boards reported that about one-fourth of youth received services primarily from WIA one-stops--whether the one-stops focused on serving adults or youth. However, among the one-stops we visited, these entities were frequently used for supplemental services such as information and referrals. Officials we spoke with in New Hampshire and Ohio said that service providers or schools usually took youth on a field trip to the local one-stop for basic orientation to expose them to employment services they could use in the future. Although most youth were served by the WIA program through other providers, one-stop usage varied considerably by local area. For example, the majority of WIA youth living in an urban area of California were served by youth- exclusive one-stops, while youth in a rural area of the state rarely interacted with the local adult one-stop. In-school and out-of-school youth usually did not participate in WIA programs together. Nationwide, 43 percent of local boards reported that these two types of youth received services from different providers. In another 27 percent of cases, the two groups received services from the same providers but participated separately. Another 24 percent of local boards reported that all youth were served by the same service providers and participated together.[Footnote 16] Local Areas Developed Partnerships with Business and Education to Deliver Youth Services: Local areas developed partnerships with the local business and education communities to identify employer needs and provide comprehensive youth services related to academic and employment preparation. In many cases, youth councils helped facilitate these partnerships. Local areas partnered with businesses to identify and provide employment and training services for WIA youth needed to fill high- demand, high-growth occupations.[Footnote 17] Over 85 percent of local boards reported using each of the following methods to develop and maintain relationships with local businesses: consulting with businesses about their job needs, training participants in skills needed by local businesses, training program participants to understand the values and attitudes local businesses look for in employees, and providing employment experience to participants that suits the jobs available in local businesses. Officials we spoke with at the local youth programs in all five states we visited provided such work readiness training as punctuality, teamwork, respect for others, and appropriate dress. In addition to identifying needed skills, businesses also provided employment opportunities. Nationally, 34 percent of local boards reported that businesses subsidized work experience for WIA youth. In New Hampshire, for example, a financial services firm employed out-of-school youth and taught them about personal financial management. In addition, an appliance-store owner, once an at-risk youth himself, hired WIA youth and hoped to groom one of them to take over the business. Finally, officials in three states we visited noted that businesses donated in-kind assistance such as building materials, work clothes, work readiness training workshops, and a financial management curriculum. For example, in an urban area in Virginia, businesses donated and remodeled the space for the one-stop center. Local areas also partnered with schools to provide academic preparation services to WIA youth. In a local area of Louisiana, for example, a service provider helped eighth graders explore classes leading to high school industry-based certifications. In all five states we visited, representatives from local postsecondary institutions made presentations to WIA youth, informing them about higher education opportunities. In some instances, WIA youth were allowed to take advanced level courses at their institutions and even earn college credit. In addition, a community college in California provided a pathway to facilitate the transition from high school to higher education by hosting youth for a one-day college experience of classes and conducting workshops on financial aid. Officials in four states we visited also said that schools provided in-kind assistance such as office space and tutoring. Local boards reported that youth councils performed a number of important functions that facilitated partnerships between local boards and the community. Over 70 percent of local boards reported that the youth council served as a forum to bring together key community partners who may never have collaborated with each other. Almost two- thirds of local boards reported that they would keep their youth council even if it became optional.[Footnote 18] In addition, over half of local boards reported that their youth councils elevated the importance and visibility of local youth issues and programs and added value to the youth program. Labor's Guidance Has Not Fully Addressed Implementation Challenges Faced by Local Areas: Local areas continue to face challenges in implementing aspects of the WIA youth program despite guidance issued by Labor. Some of these ongoing challenges include recruiting and retaining out-of-school youth, providing mentoring and follow-up services, and designing interim performance measures that can be used to continually improve aspects of program performance before youth exit the program. Labor supplements guidance on these issues through information forums, a Web site, and a technical assistance program. However, regional Labor officials said that state and local areas' access to these resources had been limited. Out-of-School Youth Were Hard to Recruit and Retain: Labor has issued guidance on recruiting and retaining out-of-school youth for the WIA program during the last two years, but local areas continue to face challenges in serving this population. Labor issued a guidance letter to states in April 2001 outlining strategies for recruiting youth to the program, suggesting such methods as engaging youth to recruit their peers, collaborating with community organizations that already work with disadvantaged youth, and offering incentives for recruiting new participants.[Footnote 19] In subsequent guidance issued in May 2002, Labor described ways to keep out-of-school youth engaged in the program, such as by helping youth make the link between career potential and education and skills training as well as by providing support services like child care.[Footnote 20] Despite this guidance, local officials from most of the states we visited said that recruiting out-of-school youth continues to be a problem. According to data from Labor, the District of Columbia, Delaware, and New Mexico had not met WIA's requirement to spend at least 30 percent of WIA funds on out-of-school youth for program year 2001 as of September 30, 2003.[Footnote 21] Some local workforce officials indicated that closer coordination with schools to immediately connect dropouts to the WIA program would help them identify and recruit more out-of-school youth. Some local workforce and regional Labor officials said that one reason schools may not share information on dropouts directly with the WIA program was because of concerns about student privacy restrictions. However, schools we visited in four states shared dropout information by referring students who had been expelled or had dropped out directly to the WIA service provider, notified or provided contact information to the WIA provider when a student had dropped out, or worked with the WIA program to develop a list of dropouts. None of the schools, however, had procedures in place to routinely connect all dropouts with local WIA programs. Labor and Education officials agreed that schools could do more to work closely with local workforce officials to help connect dropouts to local WIA youth programs. Local officials from most of the states we visited said that retaining out-of-school youth in the WIA program was also difficult. According to some local officials, one reason was that out-of-school youth tend to prefer immediate employment instead of training and academic learning. Other officials said that both in-school and out-of-school youth face difficulties with transportation--particularly in rural areas---that limit their involvement in WIA programs. For example, local officials in a rural area in Ohio said that transportation was their biggest issue. To mitigate this challenge, the local area offered services through a mobile one-stop unit that traveled to WIA clients throughout the county. Some state workforce officials said that to be more effective in serving this population, they needed guidance and technical assistance that was more focused on the specific service delivery issues within their local areas. Mentoring and Follow-up Were among the Most Difficult Services to Provide: Nearly all local areas we visited indicated that providing mentoring services continued to be a problem for the WIA youth program, but Labor has not addressed mentoring in its annual youth program guidance or shared best practices on the provision of high-quality mentoring services. Local areas identified several reasons why mentoring has been difficult. In one instance, a Virginia official in a rural area said that it was difficult to identify sufficient numbers of qualified mentors to work with eligible youth. In another instance, a local area official in California noted that geographical distances within the county resulted in long commutes and discouraged adults from mentoring youth. A service provider in New Hampshire said that finding mentors was especially difficult for out-of-school youth, as some adults are uncomfortable working with school dropouts. Labor youth program officials said that mentoring is an important way of connecting youth to caring adults and said they would take action to provide guidance on this topic. Another service element that remains challenging for local areas is providing complete and thorough follow-up services to help youth succeed after they have exited the program. WIA regulations require that follow-up services last for at least 12 months, and Labor's guidance states that follow-up may include regular contact with a youth's employer to track progress made, assistance in addressing work- related problems, and help in securing better-paying jobs and further education. Labor's policy guidance for program year 2001 provided some principles from best practices in the field of youth development to help local areas develop strategies for follow-up. The principles included developing a systematic approach for maintaining contact and interaction with the young person; evening and weekend social activities for informal support; helping youth access services to fulfill physical, emotional, and vocational needs; and visiting the job site soon after the youth has started employment. Nevertheless, several local officials cited continued difficulties in sustaining a connection with youth and identifying outcomes for them once they exited the WIA program. Few Local Areas Used Interim Measures to Gauge Program Success: Labor's guidance underscored the importance of establishing and using optional interim performance measures to monitor the success of delivering WIA youth services, but some regional Labor officials said that such measures were not widely used. Six of the seven required WIA youth measures are collected only after youth exit the program. However, because youth may remain in the program for many years, local areas with long-term youth retention strategies may have limited means of gauging progress without interim measures. Labor issued guidance in May 2002, encouraging the use of interim measures, which may be tracked and reported separately from the required annual performance measures, to provide ongoing feedback on which aspects of the youth program were working well or needed modification.[Footnote 22] The guidance identified the following interim measures that local boards could use to monitor progress as youth move through the WIA program: * intake, including development of a comprehensive individual service strategy with short-term and long-term goals; * participation in program elements to help ensure youth meet short- term skill attainment goals; * skill attainment to help ensure successful completion of short-and long-term goals; * exit data, including number of participants exiting the program; and: * follow-up services received to help measure youth performance outcomes. Some local areas we visited were using interim measures to hold providers accountable for delivering services. For example, New Hampshire workforce officials said that they tracked enrollment and expenditure levels to monitor activity levels across their contracted service providers. In a local area in California, workforce officials monitored the progress of youth by tracking enrollment, participation, work readiness skills, and the rate at which youth made a successful transition to other activities once they exited the WIA program. Despite the potential usefulness of these data, interim measures were not widely used, according to Labor officials in two regions. These officials said that states lacked the resources to properly track them. Additionally, the guidance did not explain how states and local areas could collect, analyze, and use the data to assess progress and make needed adjustments. For example, officials in one state we visited said that they were unclear about how to apply the interim measures. Access to and Awareness of Labor's Assistance Has Been Limited: Labor has disseminated information related to these and other issues through forums that allow local areas to exchange information with one another, but access to these forums has been limited. From December 2002 to April 2003, for example, Labor sponsored peer-to-peer WIA Learning Exchanges with two national youth organizations that provided venues around the country for local areas to share information and observe an exemplary program firsthand. However, state budgetary cuts prevented many local areas from attending some of these forums in person, according to an official from the sponsoring organization. Although information and ideas shared at the Learning Exchanges were later summarized and made available to all local areas, those that were unable to attend missed the opportunity to network with their colleagues and develop an action plan to take back and apply to their program.[Footnote 23] Labor has also supported the development of an online resource to facilitate information sharing among local areas. The agency's Promising Practices Web site was intended to provide a mechanism for local areas to post and share promising workforce development practices, including those pertaining to WIA youth programs. However, some state workforce and regional Labor officials said that states and local areas may not even have been aware of the site, that it has been difficult to enter practices for inclusion, and that some users found it difficult to access the documents described in the narrative.[Footnote 24] We also had difficulty accessing and using the Web site to find information. For example, when we conducted a search on the key word "mentoring," the five results did not satisfactorily address the topic. Labor guidance encourages local areas to tap into other resources that can help them develop quality youth programs.[Footnote 25] The Promising and Effective Practices Network, for example, provides a useful listing of promising practices categorized by the specific strategy and the 10 required WIA youth services that local areas can use for improving their WIA youth programs.[Footnote 26] Labor has provided technical assistance to states through its Performance Enhancement Project, initiated in program year 2002. Under this initiative, Labor grouped into three tiers states that failed or were at risk of failing to meet their performance goals, based on their reported outcomes, according to Labor officials.[Footnote 27] These officials also said that states in the first tier received priority for targeted technical assistance to improve their youth programs, with the goal of improving performance outcomes. Labor officials said that states could use this technical assistance to address any of the challenges they faced in implementing their WIA youth program. However, unless the state falls into one of these three tiers, most local areas may not receive such assistance to help them increase the proportion of out-of-school youth served, improve mentoring and follow-up, and use interim measures. Lack of Program Evaluations and Questionable Performance Data Prevent Assessments of WIA Youth Program: Little is known about the effectiveness of the WIA youth program because Labor has not yet initiated an impact evaluation, and results from a planned evaluation will likely not be available until 2009, according to Labor officials. While Labor's performance data for program year 2001 indicate that five of seven youth measures were exceeded, these results cannot be used to infer program outcomes because of insufficient monitoring of state data quality and inadequacies of some state management information systems. Labor officials said that they will require states to validate local area data beginning with program year 2002 data, but some implementation concerns remain. Labor Plans to Initiate First Impact Evaluation in 2006, and Results Are At Least Five Years Away: According to department officials, Labor intends to initiate an impact evaluation[Footnote 28] of the WIA youth program in 2006, and preliminary results will not be available until about 2009. While Labor officials said that the youth program's 3 years of full implementation was sufficient time to initiate a comprehensive evaluation, they were anticipating significant changes to the WIA youth program as a result of reauthorization. They did not plan to begin an impact evaluation of the program until after these changes had been implemented. If reauthorization is completed by spring 2004, Labor officials said they anticipate that the process of awarding the contract for the study will be completed by 2006 when the impact evaluation is scheduled to begin. The evaluation will proceed with 5 or 6 years of data collection with an additional 3 or 4 years of follow-up activity. Labor officials said they expect to issue a series of interim reports before the final product. If the project begins in 2006, Labor expects to issue the first report on short-term impacts in 2009, with a final report on long-term impacts available in 2011. According to officials, this evaluation is part of Labor's proposed research plan to study all of its major employment and training programs, including the WIA youth program, on a regular cycle.[Footnote 29] State Performance Outcome Data Were Questionable, Partly because of Inconsistent Data Monitoring: Performance data submitted by states to Labor in quarterly and annual reports were not sufficiently reliable to determine outcomes for the WIA youth program. Labor's national aggregation of WIA performance data for program year 2001 indicated that the program exceeded its goals for five of the seven youth measures.[Footnote 30] According to Labor's Office of Inspector General (OIG), however, there is little assurance that the states' performance data for all WIA programs, including the youth program, are either accurate or complete because of inadequate oversight of data collection and management at the local, state, and federal levels. The OIG also found that just 2 of 12 local areas and none of four states it reviewed had formal policies on documentation requirements for participant activities and outcomes.[Footnote 31] At the local level, for example, another OIG report found that WIA youth program outcomes were adequately documented only 37 percent of the time for a sample of 420 participants across 14 local areas.[Footnote 32] An official in one of Labor's regional offices added that documentation requirements are inconsistent among states and local areas. At the state level, the OIG reported that two of four states it reviewed had not monitored local areas' performance data at the case file level, and that none of the four states had adequate procedures in place to ensure the accuracy of their performance data.[Footnote 33] According to regional Labor officials, some states had insufficient procedures in place for verifying the accuracy of the data collected by their local areas. For example, officials in one region said that state monitoring rarely addressed data accuracy or included the verification of a sample of data items against original records. In addition, some state information systems had significant flaws that caused them to produce incorrect information, casting further doubt on the quality of WIA performance data. For example, Ohio state officials said they were in the process of replacing their old information system that had caused such errors as data changing or disappearing entirely after it had been entered in the system. At the federal level, Labor did not have a standard data-monitoring guide in place, and officials in some regional offices--who, according to agency officials, are responsible for overseeing the quality of states' reported data---said they followed their own oversight procedures. These procedures did not usually include verifying the accuracy of a sample of the data submitted by states. For example, an OIG report stated that while regional offices conducted some data accuracy reviews, such as computer edit checks, they did not verify the data's accuracy with such tests as comparing the data with participant case files. Consistent with the OIG's findings, officials in all six of Labor's regional offices said that they examined state data submissions through desk reviews, which included checking for errors such as incorrect calculation of performance measures, extreme outliers, and miscoding of data. However, only the Atlanta regional office checked a sample of data records against source documentation. In its review of data records from six of the states it oversees, the Atlanta office examined a sample of participant records across all WIA programs from each state and found errors that affected the calculation of one or more performance measures. In one state, for example, 17 percent of participant records had at least one error, compared with 83 percent of participant records in another state. The regional office also found that two of six states it reviewed computed the younger youth skill attainment measure incorrectly. Labor recognizes these problems with data reliability, but in accordance with WIA regulations, uses states' annual performance reports---the only reports that depict states' progress in achieving or exceeding negotiated performance levels---to make incentive grant and sanction decisions. States that exceed the WIA performance goals negotiated between Labor and the states are eligible to receive incentive grants that generally range from $750,000 to $3 million.[Footnote 34] States that fail to meet 80 percent of their WIA performance goals for 2 years in a row are subject to sanction of up to a 5 percent reduction in their annual WIA formula grant. However, Labor's use of questionable performance data reported by the states may hinder its ability to negotiate realistic performance goals and make appropriate incentive grant and sanction decisions. Ohio state officials questioned the appropriateness of being sanctioned 2 percent of its program year 2002 WIA youth allotment. They believed that poor performance levels were due to problems with its information system, not its workforce development system. Data Validation Initiative May Improve Reliability but Concerns Remain: To address the data issues described above, Labor is implementing a new data validation policy requiring states to ensure the accuracy of their annual reports and verify a sample of the data they submit. According to Labor's policy, these requirements will be phased in over a 3-year period. In the first year, states will be required to validate their annual reports and data submissions for the program year 2002 period.[Footnote 35] According to agency officials, Labor does not plan to publicly release these initial reports but will use their findings to work with the states to correct their data accuracy problems. In the second year, Labor will use validation reports covering program year 2003 data to establish acceptable error rates. In the third year, Labor will require states to meet the acceptable error rates for their program year 2004 data submissions. Labor will consider failure to meet the standard a violation equivalent to failing to submit a report, for which states may be subject to corrective action or financial sanction, as appropriate. In addition, Labor's data validation guidance indicates that states that do not meet data accuracy standards will receive technical assistance. While the data validation initiative may improve the reliability of WIA performance data, several implementation concerns remain. First, officials in some of Labor's regional offices said that the states they oversee will have difficulty implementing the data element verification requirement because of limited staff resources. Further, these officials said that in states where local providers keep the original documentation on-site, retrieving the documentation to check it against records in their information system will be time-consuming and expensive. Second, a Labor official said that the agency does not plan to issue a program-monitoring guide to standardize procedures across regional offices for several more years. Conclusions: Many youth are struggling to be successful in the public education system and often face substantial obstacles to obtaining a high school diploma and going on to college or getting jobs with career advancement possibilities. Many of these youth may lack meaningful social and family supports and may require comprehensive, intensive services to remain engaged in society and avoid risky behaviors that can lead to chronic unemployment, criminal activity, and other adverse outcomes. WIA currently provides both the education and workforce systems with strategies and resources to engage youth in academic and job training. The opinion held by some federal education officials that WIA in-school services overlap with existing education programs is not necessarily seconded by officials in local areas. State and local workforce and education officials believe WIA's educational, occupational, and support services provide critical support and services to at-risk youth and that without WIA's comprehensive services, schools may face an increasing burden to keep these youth in school and ensure their academic success. The connection between WIA youth services and schools could be made more effective if Labor worked with Education to find ways to connect school dropouts with local WIA youth programs. Since WIA's passage, Labor has provided general guidance and promising practices in addressing implementation issues, but increased availability of technical assistance may be necessary to overcome some of the more difficult challenges some states and local areas face in providing youth services. This will be especially critical for states and local areas that are reportedly meeting performance goals but still need assistance in improving delivery of youth services and shifting program resources to target different populations such as out-of-school youth. In addition, local areas will need guidance, including specific strategies, to help them provide effective mentoring and follow-up services and use interim measures to track program performance. Robust research and reliable performance data are needed to obtain a complete picture of the WIA youth program's effectiveness and outcomes and to make quality decisions about managing the program. However, none currently exist. Labor has not yet initiated an impact evaluation and does not expect to report on program impacts from a planned evaluation until 2009. In addition, states and local areas continue to struggle to collect and document accurate and complete participant and performance data and maintain data systems that can yield reasonably reliable outcome information. For its part, Labor is taking action to improve data integrity by requiring states to validate WIA performance data, beginning with data from program year 2002. However, Labor's inconsistent monitoring processes across regions will continue to challenge Labor's capability to ensure that all states validate and report data consistently and effectively. In the short term, the lack of accurate outcome data will continue to hinder Labor's ability to negotiate realistic state performance goals and use data to make sound decisions about financial incentives or sanctions. In the long term, lack of accurate and complete information will keep Labor from obtaining a true picture of how effectively the youth program is working. Recommendations for Executive Action: To promote information sharing that improves local WIA youth programs' ability to identify and serve out-of-school youth, we recommend that the Secretaries of Labor and Education coordinate efforts to clarify how schools can work with workforce officials to connect school dropouts with local WIA youth programs. To assist state and local WIA youth programs address ongoing implementation challenges, we recommend that the Secretary of Labor: * increase availability of guidance and technical assistance to local areas that continue to face challenges in serving out-of-school youth; * disseminate guidance, including specific strategies, to help local areas provide effective mentoring services; and: * develop additional guidance on providing follow-up services and using interim measures to track program performance. To obtain an accurate and complete gauge of WIA outcomes and determine whether local programs are operating successfully, we recommend that the Secretary of Labor establish standard monitoring procedures that Labor's regional offices could use to oversee state data validation efforts. Agency Comments and Our Evaluation: We provided a draft of this report to Labor and Education officials for their review and comment. Education's comments are reprinted in appendix I. In its formal comments, Education concurred with our recommendation to work with Labor to connect out-of-school youth to local WIA youth programs. Labor responded informally, and said that many of the findings corroborated its own observations and that the recommendations are consistent with Labor's current program direction. Both agencies also provided technical comments, which we incorporated as appropriate. We will send copies of this report to the Secretaries of Labor and Education, relevant congressional committees, and other interested parties. Copies will be made available to others upon request. In addition, the report will be available at no charge on GAO's Web site at http://www.gao.gov. Please contact me at (415) 904-2272 if you or your staff have any questions about this report. Other major contributors to this report are listed in appendix II. Sincerely yours, Signed by: David D. Bellis: Director, Education, Workforce, and Income Security Issues: [End of section] Appendix I: Comments from the Department of Education: UNITED STATES DEPARTMENT OF EDUCATION: OFFICE OF VOCATIONAL AND ADULT EDUCATION: THE ASSISTANT SECRETARY: JAN 2 8 2004: Mr. David D. Bellis Director: Education, Workforce, and Income Security Issues United States General Accounting Office: 441 G Street, N.W. Washington, DC 20548: Dear Mr. Bellis: Thank you for the opportunity to review and comment on the draft report, Workforce Investment Act: Labor Actions Can Help States Improve Quality of Performance Outcome Data and Delivery of Youth Services (GAO-04-308). We appreciate your interest in improving the effectiveness of Federal programs that serve disadvantaged youth. In 2000, nearly 11 percent of young adults ages 16 through 24 were not enrolled in school and had not earned a high school diploma or equivalent credential. Without further education and training, these 3.8 million young people face a grim future in an economy that places a premium on education. Young adults who lack a high school credential are more likely to be unemployed and to be out of work for longer periods of time than their more educated peers. When they do secure employment, they earn significantly less and are more likely to be dependent on public assistance. Today, most jobs that pay family- supporting wages require not only a high school credential but the completion of further education and/or training. We concur with your recommendation that the Departments of Education and Labor work together to find new ways to connect more of these young people with the education and training they need to achieve independence and succeed in the workforce. During Fiscal Year 2001, local programs funded by the Adult Education and Family Literacy Act (AEFLA) served more than 1.1 million young adults between the ages of 16 and 24 who were not enrolled in school and had not earned a high school diploma or equivalent credential. However, AEFLA supports academic instruction only. The President's proposals for the reauthorization of AEFLA and Title I of the Workforce Investment Act will expand access by these young adults to occupational skills training, job placement, and other services available through the Department of Labor's One-Stop Career Center System, as well as make it easier for adult education participants to transition to postsecondary education upon earning a high school credential. The two Departments worked closely together in developing the President's proposals and we expect to continue our collaboration in implementing the new legislation upon its enactment. One of the Department's chief implementation priorities is developing a coordinated strategy for serving out-of-school youth and strengthening the connections between adult education programs and the One-Stop Career Center System. By working together, the two Departments can help communities use Federal resources more effectively to improve outcomes for out-of-school youth and ensure that no child is left behind. Thank you for the opportunity to address this important issue. Sincerely, Signed by: Susan Sclafani: Assistant Secretary for Vocational and Adult Education: [End of section] Appendix II: GAO Contacts and Staff Acknowledgments: GAO Contacts: Lacinda Ayers, Assistant Director (206) 654-5591 Meeta Sharma, Analyst- in-Charge (206) 287-4806: Staff Acknowledgments: In addition to the individuals mentioned above, the following staff made key contributions to this report: Karyn Angulo, Susan Baker, Andrew Bauck, Paula Bonin, Maya Chakko, Keira Dembowski, Joel Grossman, Corinna Nicolaou, Rebecca Woiwode, and Monica Wolford. [End of section] Related GAO Products: Transportation-Disadvantaged Populations: Some Coordination Efforts Among Programs Providing Transportation Services, but Obstacles Persist. GAO-03-697. Washington, D.C.: June 30, 2003. Workforce Investment Act: One-Stop Centers Implemented Strategies to Strengthen Services and Partnerships, but More Research and Information Sharing is Needed. GAO-03-725. Washington D.C.: June 18, 2003. Workforce Investment Act: Issues Related to Allocation Formulas for Youth, Adults, and Dislocated Workers. GAO-03-636. Washington, D.C.: April 25, 2003. Workforce Investment Act: States' Spending Is on Track, but Better Guidance Would Improve Financial Reporting. GAO-03-239. Washington, D.C.: November 22, 2002. Workforce Investment Act: Youth Provisions Promote New Service Strategies, but Additional Guidance Would Enhance Program Development. GAO-02-413. Washington, D.C.: April 5, 2002. Workforce Investment Act: Improvements Needed in Performance Measures to Provide a More Accurate Picture of WIA's Effectiveness. GAO-02-275. Washington, D.C.: February 1, 2002. School Dropouts: Education Could Play a Stronger Role in Identifying and Disseminating Promising Prevention Strategies. GAO-02-240. Washington, D.C.: February 1, 2002. FOOTNOTES [1] Center for Law and Social Policy, Leave No Youth Behind: Opportunities for Congress to Reach Disconnected Youth, (Washington, D.C.: July 2003). [2] U.S. General Accounting Office, Workforce Investment Act: Youth Provisions Promote New Service Strategies, but Additional Guidance Would Enhance Program Development, GAO-02-413 (Washington, D.C.: April 5, 2002). [3] A youth is considered to face employment barriers if he or she is (1) deficient in basic literacy skills; (2) a school dropout; (3) homeless, a runaway, or a foster child; (4) pregnant or a parent; (5) an offender; or (6) an individual who requires additional assistance to complete an educational program or to secure and hold employment. Up to 5 percent of WIA youth participants are not required to meet the income eligibility requirements, but all youth served by WIA must meet barrier requirements. [4] Out-of-school youth do not include youth enrolled in alternative schools at the time of WIA registration. [5] For a discussion of WIA formula funding, see U.S. General Accounting Office, Workforce Investment Act: Issues Related to Allocation Formulas for Youth, Adults, and Dislocated Workers, GAO-03-636 (Washington, D.C.: April 25, 2003). [6] To create a more comprehensive workforce investment system, WIA requires states and localities to bring together 17 federally funded employment and training services into a single system, called the one- stop center system. Locally, the one-stop system must include at least one physical site offering a comprehensive array of WIA services as well as those of the other partners, and may be supplemented by satellite sites that provide one or more WIA services. For more information on the one-stop system, see U.S. General Accounting Office, Workforce Investment Act: One-Stop Centers Implemented Strategies to Strengthen Services and Partnerships, but More Research and Information Sharing Is Needed, GAO-03-725 (Washington, D.C.: June 18, 2003). [7] Local boards in Alaska, Arizona, Arkansas, California, Colorado, the Federated States of Micronesia, Florida, Idaho, Louisiana, Missouri, New Jersey, New York, Puerto Rico, Tennessee, Texas, and Wisconsin reported having one or more youth-only one-stop centers. See GAO-02-413. [8] WIA Learning Exchanges for Youth Systems are two-day regional meetings designed to highlight promising practices at exemplary WIA sites, foster peer-to-peer information sharing, and develop team-driven action plans in three areas: (1) recruiting and retaining out-of-school youth; (2) building and sustaining partnerships, especially with the education community; and (3) defining and aligning assessments, skill achievement, and credentials. Along with the National Youth Employment Coalition and the American Youth Policy Forum, Labor conducted seven learning exchanges around the country, one of which was specifically for rural local areas. [9] Excluding the territories, Labor's data show that only eight states had spent 50 percent or more of their program year 2001 WIA youth allotment on out-of-school youth as of September 30, 2003: Iowa, Idaho, Illinois, Maryland, Maine, New Hampshire, North Carolina, and South Dakota. [10] For a discussion of Education and other federal programs to assist youth at risk of dropping out of school, see U.S. General Accounting Office, School Dropouts: Education Could Play a Stronger Role in Identifying and Disseminating Promising Prevention Strategies, GAO-02-240 (Washington, D.C.: February 1, 2002). [11] One of the 10 required services, follow-up, is required for all youth for at least 12 months after they exit the WIA program. [12] Research has shown that engaging youth during the summer provides them with an opportunity to learn new skills that they can apply in school and also learn the value of work. See Westat, The 1993 Summer Youth Employment and Training Program, (Rockville, MD, April 1994). In addition, Labor's guidance highlights work-based learning during summer employment as a strategy to help out-of-school youth acquire work experience while making the connection between staying in school and pursuing a career. See U.S. Department of Labor, Training and Employment Guidance Letter No. 3-99, (Washington, D.C., 2000). [13] See U.S. Department of Labor, Training and Employment Guidance Letter No. 18-00, (Washington, D.C., 2001). [14] See Training and Employment Guidance Letter No. 18-00. [15] See U.S. Department of Labor, Training and Employment Guidance Letter No. 9-00 (Washington, D.C., 2001). [16] The remaining 6 percent of local boards reported their arrangement for service delivery to in-school and out-of-school youth as "other." [17] Over 70 percent of local boards reported that their programs encouraged youth to seek employment in each of the following sectors: retail, hospitality and food service, health care, and information technology. [18] Of the local boards that reported that they would not keep their youth council, 73 percent reported that they would use a youth committee of the local board to perform the functions of a youth council. [19] See Training and Employment Guidance Letter No. 18-00. [20] See U.S. Department of Labor, Training and Employment Guidance Letter No. 28-01, (Washington, D.C., 2002). [21] Under WIA, states have three years to spend their annual allotment, therefore program year 2001 funds must be expended by June 30, 2004. [22] See Training and Employment Guidance Letter No. 28-01. [23] Labor officials told us that their regional offices have also sponsored youth conferences and other forums where guidance and technical assistance have been provided. [24] We previously reported on problems with the Promising Practices Web site. See GAO-03-725. This Web site can be found at http:// www.promising-practices.org. [25] See Training and Employment Guidance Letter No. 18-00. [26] The National Youth Employment Coalition's Promising and Effective Practices Network Web site can be found at http://www.nyec.org/pepnet. [27] According to Labor, the first tier includes those states that were financially sanctioned. The second tier includes those states that failed one or more performance measures but were not sanctioned. The third tier includes those states that did not fail any measures but were deemed at risk of failing them. [28] By isolating a program's effects, impact evaluations provide policymakers with key information for determining program effectiveness. To isolate a program's effect, impact evaluations divide participants into two groups: one that receives program services and a similar group that does not. Some impact evaluations assign participants randomly to one or the other group; the group that does not receive services is called the control group. The use of random assignment allows researchers to compare outcomes for the two groups and attribute any differences to the program services rather than other factors. [29] Labor anticipates that the WIA youth program will be evaluated on a 10-year cycle. [30] States did not meet goals for the earnings change measure and credential rate for older youth. States reported actual earnings change in 6 months as $3,109, compared with their negotiated goal of $3,396. For the credential rate, about 40 percent of older youth earned credentials, compared with the goal of 44 percent. [31] U.S. Department of Labor, Office of Inspector General, Workforce Investment Act Performance Outcomes Reporting Oversight, 06-02-006-03- 390 (Washington, D.C., Sept. 30, 2002). [32] U.S. Department of Labor, Office of Inspector General, Workforce Investment Act: Evaluation of Youth Program Enrollments, Services, and Recorded Outcomes, 06-03-006-03-390 (Washington, D.C., September 30, 2003). [33] U.S. Department of Labor, Office of Inspector General, 06-02-006- 03-390. [34] To be eligible for incentive grants, states must also meet performance goals for the Adult Education and Literacy programs under Title II of WIA and programs authorized under the Carl D. Perkins Vocational and Applied Technology Education Act. [35] Program year 2002 started on July 1, 2002, and ended on June 30, 2003. GAO's Mission: The General Accounting Office, the investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through the Internet. GAO's Web site ( www.gao.gov ) contains abstracts and full-text files of current reports and testimony and an expanding archive of older products. The Web site features a search engine to help you locate documents using key words and phrases. You can print these documents in their entirety, including charts and other graphics. Each day, GAO issues a list of newly released reports, testimony, and correspondence. GAO posts this list, known as "Today's Reports," on its Web site daily. The list contains links to the full-text document files. To have GAO e-mail this list to you every afternoon, go to www.gao.gov and select "Subscribe to e-mail alerts" under the "Order GAO Products" heading. Order by Mail or Phone: The first copy of each printed report is free. Additional copies are $2 each. A check or money order should be made out to the Superintendent of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a single address are discounted 25 percent. Orders should be sent to: U.S. General Accounting Office 441 G Street NW, Room LM Washington, D.C. 20548: To order by Phone: Voice: (202) 512-6000: TDD: (202) 512-2537: Fax: (202) 512-6061: To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov Automated answering system: (800) 424-5454 or (202) 512-7470: Public Affairs: Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800 U.S. General Accounting Office, 441 G Street NW, Room 7149 Washington, D.C. 20548: