This is the accessible text file for GAO report number GAO-14-363T entitled 'VA Vocational Rehabilitation and Employment: Further Program Management Improvements Are Needed' which was released on February 27, 2014. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. United States Government Accountability Office: GAO: Testimony: Before the Subcommittee on Economic Opportunity, Committee on Veterans' Affairs, House of Representatives: For Release on Delivery: Expected at 10:00 a.m. EST: Thursday, February 27, 2014: VA Vocational Rehabilitation and Employment: Further Program Management Improvements Are Needed: Statement of Daniel Bertoni, Director: Education, Workforce and Income Security: GAO-14-363T: Chairman Flores, Ranking Member Takano, and Members of the Subcommittee: I am pleased to discuss our work on the Department of Veterans Affairs' (VA) Vocational Rehabilitation and Employment (VR&E) program. The VR&E program helps veterans with service-connected disabilities [Footnote 1] obtain and maintain suitable employment, which VA defines as employment consistent with veterans' abilities and interests. Eligible veterans may receive services such as vocational assessment, education, training, and job placement.[Footnote 2] In fiscal year 2012, about 121,000 veterans participated in the program at a cost of almost $1 billion. The program is administered by VA staff located in the agency's 57 regional offices or additional satellite locations. In 2009, we reported on a number of challenges VA faced managing this program. More recently, the VOW to Hire Heroes Act of 2011 (VOW Act) mandated that GAO review the VR&E program.[Footnote 3] My remarks today are based on the report issued January 14, 2014, which examined: (1) the outcomes for veterans seeking employment through the VR&E program, and (2) the progress VA has made in addressing critical management issues.[Footnote 4] To inform our work, we reviewed relevant federal laws, regulations, guidance, and documentation related to recent management initiatives; analyzed VA administrative data on VR&E participants who applied to the program between fiscal years 2003 and 2012;[Footnote 5] and interviewed central and regional office staff responsible for administering the VR&E program and providing services.[Footnote 6] We also interviewed a nongeneralizable sample of 17 veterans who had participated in the VR&E program, and reviewed methods and results associated with VA's pilot satisfaction survey of VR&E participants. We conducted this work from October 2012 to January 2014 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. A more detailed explanation of our methodology is available in our report. With respect to program outcomes, we found that of almost 17,000 veterans who entered the VR&E program in fiscal year 2003--the earliest year for which reliable data are available--and received employment-related services, just under half had achieved rehabilitation (that is, suitable employment) by the end of fiscal year 2012. Another 30 percent had discontinued their participation in the program, although some of these were considered by VA to have obtained some benefits from program participation even though they did not achieve suitable employment.[Footnote 7] Most of the rest were still receiving program services. (See figure 1.) Figure 1: Outcomes to Date for Fiscal Year 2003 Applicants Who Received Employment-Related Services: [Refer to PDF for image: pie-chart and expanded portion of pie-chart] Placed in suitable job: 47%; Still participating: 20%; Achieved independent living goal: 2%; Discontinued: 30%: - Maximum rehabilitation gain (MRG): 30%; - Other reason: 70%. Source: GAO analysis of VA data. Note: The percentages in the four outcome categories do not add up to 100 percent due to rounding. It is possible that some veterans who discontinued in fiscal year 2010 or later are included as MRG cases in this figure even though they actually discontinued for a different reason. This is because of a change in fiscal year 2010 in the meaning assigned to certain discontinuation reason codes in VA data. [End of figure] We also found that while the average time to achieve employment was about 4 ½ years, time frames varied significantly. About half the veterans achieved success within 3 to 6 years, but many took considerably longer (see figure 2). One reason for lengthy time frames is that some ultimately successful participants discontinued or interrupted[Footnote 8] their programs along the way. Of those who achieved employment, about one-quarter interrupted their programs at least once, and about the same proportion discontinued, then returned to the program (some more than once) before ultimately achieving success. Figure 2: Time to Achieve Suitable Employment for Fiscal Year 2003 Applicants: [Refer to PDF for image: combined vertical bar and line graph] Years since application: 1; Percentage (and number) of veterans[A]: 3%; 214; Cumulative percentage of veterans[B]: 3%. Years since application: 2; Percentage (and number) of veterans[A]: 10%; 749; Cumulative percentage of veterans[B]: 13%. Years since application: 3; Percentage (and number) of veterans[A]: 15%; 1,138; Cumulative percentage of veterans[B]: 27%. Years since application: 4; Percentage (and number) of veterans[A]: 17%; 1,290; Cumulative percentage of veterans[B]: 44%. Years since application: 5; Percentage (and number) of veterans[A]: 17%; 1,318; Cumulative percentage of veterans[B]: 61%. Years since application: 6; Percentage (and number) of veterans[A]: 14%; 1,094; Cumulative percentage of veterans[B]: 76%. Years since application: 7; Percentage (and number) of veterans[A]: 9%; 730; Cumulative percentage of veterans[B]: 85%. Years since application: 8; Percentage (and number) of veterans[A]: 7%; 557; Cumulative percentage of veterans[B]: 92%. Years since application: 9; Percentage (and number) of veterans[A]: 5%; 409; Cumulative percentage of veterans[B]: 98%. Years since application: 10; Percentage (and number) of veterans[A]: 2%; 187; Cumulative percentage of veterans[B]: 100%. Source: GAO analysis of VA data. Note: Percentages of veterans placed in each year do not add up to 100 due to rounding. [A] Indicates the percentage of successfully rehabilitated fiscal year 2003 applicants placed in suitable employment in each year following application. [B] Indicates the cumulative percentage of successfully rehabilitated fiscal year 2003 applicants placed in suitable employment by each year following application. [End of figure] We identified several common challenges that VR&E participants face to successfully completing their programs and achieving rehabilitation. Specifically: * Our analysis of VA data indicates that VR&E participants with mental health conditions were 12 percentage points less likely to achieve suitable employment within 8 years of program entry compared to participants without such conditions, the largest difference for any type of disability.[Footnote 9] * Our interviews with VA staff and veterans indicate that VR&E participants face challenges with developing employment plans that can realistically lead to success. For example, VA staff in one office said that veterans with mental health conditions may want to pursue an unrealistic career in law enforcement. Veterans we interviewed also cited challenges in reaching agreement with VR&E staff on an employment plan. Our analysis of VA data indicates that program participants often stop their program then return to receive additional evaluation services. * Our analysis of VA data indicates that VR&E participants who work with more VR&E staff over time are less likely to achieve suitable employment. For example, veterans who worked with four staff were 27 percentage points less likely to achieve success within 8 years of program entry, compared to those who worked with only one staff member.[Footnote 10] * Our analysis of VA's satisfaction survey results indicates that many veterans who discontinued or interrupted their employment programs cited "family obligations" (27 percent) or "financial difficulties" (18 percent) as a reason.[Footnote 11] VR&E managers and staff also noted such challenges; for example, managers in one office said veterans may quit training or take a job not suitable for their disability because they need to support their families. * Managers, staff, and veterans we interviewed cited challenges with the transition from military to civilian employment. For example, managers or staff told us veterans may have difficulty translating military experience into terms that civilian employers can understand, or that they may need civilian credentials to get a job even with the expertise they gained in the military. Once veterans achieve suitable employment, few return for additional services.[Footnote 12] Specifically, we found that, of those who were rehabilitated, only 4 percent applied for and 2 percent ultimately returned to VR&E to receive additional employment-related services. Further, at the time of our review, only a small number of rehabilitated veterans returned to receive additional services under the expanded eligibility criteria authorized by the VOW Act.[Footnote 13] VR&E managers or staff told us almost all the veterans who are eligible for additional services under the expanded criteria already qualify under the old criteria, which provide more months of additional services. With regard to VA's progress in addressing critical VR&E management issues, we found that weaknesses remain in performance management, workload management, and staff training, although the agency has made efforts to improve each of these areas. VA has an initiative underway to improve its primary measure of program performance, but the alternative measure under consideration falls short of good practices. VA's primary measure of performance is its rehabilitation rate, which is the number of successful rehabilitations divided by the total number of successful rehabilitations and unsuccessful case closures. VA central and regional office staff cited problems with the current VR&E rehabilitation rate measure, including that it encourages VR&E staff to delay or avoid closing their cases unsuccessfully because this counts against their rehabilitation rate performance measure. To address these and other issues, VA is revising its individual, regional, and program-wide performance measures and plans to move from a rehabilitation rate measure for assessing individual staff performance to a measure of the number of positive outcomes, including rehabilitations. While this approach may be appropriate for assessing individual performance, as a measure of program performance, it lacks transparency because it does not reflect the extent to which veterans are discontinued from the program without rehabilitation. Further, this measure does not distinguish between rehabilitations and other benefits attained from participation--such as when the veteran gains some vocational skills from participation but does not obtain and maintain suitable employment. To provide a fuller picture of program success, we recommended that VA include in any revised set of national and regional measures (1) a measure of the proportion of participants successfully rehabilitated to employment, and (2) a measure of the proportion of participants who attained other benefits from participation. VA concurred with this recommendation and noted that rate-based measures will be included in its revised set of national VR&E performance measures. In addition, VA lacks and does not intend to develop performance measures related to veterans' long-term employment outcomes, even though it is planning to collect information that could serve as a basis for such measures. Part of VR&E's mission is to help veterans maintain employment, and to address this, VA counts a job placement as a successful rehabilitation if it is maintained for at least 60 days. [Footnote 14] However, research and practices at other federal employment programs suggest that tracking employment for 60 days may not be sufficient to gauge whether a veteran will maintain a job for the long term. For example, Department of Labor employment and training programs measure job retention over 180 days. VA is considering contacting rehabilitated and discontinued VR&E participants at 6 and 12 months after they finish the program to identify veterans who are having difficulties in maintaining employment, and possibly provide them additional supports. However, VA does not plan to use this information to develop a performance measure of longer-term employment outcomes, and thus may miss an opportunity to hold staff accountable for and make program adjustments to improve such outcomes. We recommended that VA develop new measures of long- term employment success that go beyond the current 60-day minimum, possibly drawing on its planned post-exit surveys. VA concurred with this recommendation in principle. However, it noted that because VA staff typically work with VR&E participants for years while in the program and potentially for up to 18 months following training completion, it sees no value in devoting limited resources to additional post-placement monitoring. We continue to believe that VA needs to hold itself accountable for the longer-term employment success of all participants, and that this could be done in a cost- effective way. Lastly, with respect to performance management, although VA has generally followed recognized practices in designing its customer satisfaction survey, the reliability of pilot results is uncertain. Specifically, we found the agency has generally followed recognized survey practices including detailed survey planning, comprehensive questionnaire development and testing, and extensive data collection activities. However, VA did not perform a nonresponse bias analysis of the pilot survey results, even though response rates for the pilot surveys were below 30 percent. Guidance issued by the Office of Management and Budget calls for nonresponse analysis when a survey's unit response rate is below 80 percent.[Footnote 15] Since VA plans to use survey results to inform program improvements and training developments, to ensure that this survey effort yields reliable data to drive program improvements, we recommended that--as warranted by response rates--VA conduct nonresponse analysis of its ongoing customer satisfaction survey results. VA concurred with this recommendation and noted that if funding permits, it will modify its survey administration contract to include such an analysis. With regard to workload management, problems with VA's formula for allocating staff among regional offices may be contributing to some offices' ongoing workload challenges. VA has taken steps to address workload management issues, such as reducing paperwork requirements. However, some offices we contacted still have concerns about high workloads, and the ratio of VR&E cases to staff varies considerably among the regions. In March 2013, for example, seven offices averaged fewer than 100 cases per staff member, while eight averaged over 175. VA's formula for allocating staff among the offices--which has not been significantly revised since fiscal year 2003--is based on offices' workloads and their performance on several metrics. Regional managers in five of the eight offices we contacted raised concerns about the formula, indicating that it may not effectively assign staff where they are most needed. For example, one concern is that the formula does not take into account educational and vocational counseling cases, which according to VA data may be disproportionately concentrated in certain offices. Another concern is that by assigning more staff to higher-performing offices, the formula may make it difficult for lower-performing offices to improve. To the extent the formula does not align staffing resources where needed, it may reduce some offices' ability to effectively serve veterans. We recommended that in revisiting its allocation formula, VA should consider these two issues and make adjustments as needed. VA concurred with this recommendation and said it plans to revisit its formula. VA also lacks information about regional offices' varied approaches to managing their caseloads. VA's administration of the VR&E program is decentralized, and according to VA central office officials each office's management has discretion about how to manage their caseloads and assign cases among staff. Some offices, for example, have veterans work with one staff member throughout their participation in the program, citing as an advantage the rapport that develops between veteran and staff member. Others have their staff specialize in particular phases of the rehabilitation process, such as evaluation or case management. Although veterans work with different staff members, some managers and staff said this approach leads to efficiencies. VA's central office, however, has not collected information on or assessed the effectiveness of the regional offices' varied approaches. Further examination of these approaches is especially important given our finding that veterans who work with more VR&E staff over time are less likely to achieve suitable employment. We recommended that VA collect information on the regional offices' approaches for managing their VR&E workloads, assess their advantages and disadvantages, and use the results of this assessment to provide guidance to the offices. VA concurred with this recommendation and noted steps it plans to take to address it. In the area of staff training, some gaps remain in its curriculum for VR&E staff, although VA has taken steps to improve training. In recent years, VA added new courses for VR&E staff, which has reduced redundancy in training from year to year and closed a number of gaps identified in a 2010 skill gap analysis. However, based on our review of the VR&E training curriculum and comments from regional office managers and staff, we found that VA does not provide sufficient training on strategies for helping veterans with job placement and workplace accommodations, which are key to ensuring veterans with disabilities successfully obtain and maintain employment. In particular, such training is not part of the curriculum for VR&E staff whose main functions are assessment and case management but who may also assist participants with job placement. We recommended that VA provide additional training to staff on job placement and workplace accommodations. VA concurred with this recommendation and said it plans to develop new courses in these areas. Chairman Flores, Ranking Member Takano, and Members of the Subcommittee, this concludes my prepared remarks. I would be happy to answer any questions that you or other members of the subcommittee may have. For further information regarding this testimony, please contact Daniel Bertoni at (202) 512-7215, or at bertonid@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Individuals who made key contributions to this testimony include Michele Grgich (Assistant Director), Jessica Botsford, David Chrisinger, Kirsten B. Lauber, and Lorin Obler. [End of section] Footnotes: [1] To qualify for VR&E services, veterans generally must have a service-connected disability rated at 20 percent or higher and an employment handicap resulting in substantial part from the service- connected disability, and typically must receive the services within either 12 years of either discharge from the military or receipt of a VA disability rating. Veterans with a 10 percent disability rating may also be eligible if they have a serious employment handicap. 38 U.S.C. §§ 3102 and 3103. [2] Eligible veterans for whom a vocational goal is not currently considered reasonably feasible may receive assistance intended to help them achieve independent living. Independent living benefits can include counseling, assistive device, and other services or equipment.38 U.S.C. § 3109. [3] Pub. L. No. 112-56, § 233(d), 125 Stat. 711, 720. [4] See GAO, VA Vocational Rehabilitation and Employment: Further Performance and Workload Management Improvements Are Needed, [hyperlink, http://www.gao.gov/products/GAO-14-61] (Washington, D.C.: January 14, 2014). [5] We assessed the reliability of these data through electronic testing and interviews with VA staff, and found them sufficiently reliable for our reporting purposes. [6] We interviewed VR&E managers and staff in 8 VA regional offices, selected to achieve diversity in VR&E caseload, performance in job placement and case processing timeliness, state unemployment rate, and geographic location. [7] VA classifies some veterans who discontinue from the VR&E program as having achieved maximum rehabilitation gain (MRG), meaning they attained vocational skills or some other benefit from program participation even though they did not achieve suitable employment. [8] According to VA, VR&E uses the interrupted status as a tool to allow veterans time to resolve unexpected issues without using up time- limited employment services. [9] We chose a period of 8 years for this analysis because many veterans take that long to achieve a successful outcome. This period of analysis provided enough time to observe successful outcomes and included results for veterans who applied in both fiscal years 2003 and 2004. We also analyzed the associations between specific disabilities and the likelihood and rate of achieving program success at various points in time after application, while controlling for other factors that may affect outcomes. Even after controlling for these factors, we still found that having a mental health condition was associated with a smaller chance of success. [10] We analyzed the association between multiple staff and the likelihood and rate of achieving program success at various points in time following application, while controlling for other factors that may affect outcomes. Among other things we controlled for the number of times veterans discontinued from and re-entered the VR&E program, which could be associated with the number of staff they worked with. Even after controlling for these factors, we still found that working with multiple staff was associated with a smaller chance of success. [11] VA's Voice of the Veteran survey measures veterans' satisfaction with benefits and services received, including through the VR&E program. The results reported here are from the pilot survey fielded from October 2012 to January 2013. The results on reasons for withdrawing or interrupting from the program are based on the responses of almost 200 veterans who had withdrawn or interrupted after spending at least 60 days in the program. We assessed the methodology of the pilot survey and concluded that the results we are reporting are sufficiently reliable for our purposes. However, these results reflect the opinions of survey respondents and cannot be generalized to the broader population of VR&E participants. [12] Under certain circumstances, veterans who have achieved suitable employment may return for additional VR&E services, such as if their disabilities have worsened to the extent they can no longer perform the occupation for which they were rehabilitated. The VOW Act expanded these eligibility criteria to include certain veterans who have exhausted their state unemployment compensation benefits. Pub. L. No. 112-56, § 233(a)(1), 125 Stat. 711, 719, (codified at 38 U.S.C. § 3102(b)). [13] According to VA, between May 2012 and April 2013 10 veterans nationwide received additional services under the new VOW Act criteria. [14] VR&E staff have discretion to track employment for a longer period if necessary, but we found that in the majority of successful rehabilitations since fiscal year 2003 (almost two-thirds), employment was tracked for less than 6 months before case closure. [15] Nonresponse occurs when members of the sample do not participate in the survey at all (unit nonresponse), or do not answer particular questions (item nonresponse). Nonresponse may result in nonresponse bias--systematic errors that result in under-or overestimation of a true value in survey results--because nonrespondents may have provided substantially different answers than those who did respond. A nonresponse analysis is an established survey research practice used to determine whether nonresponse bias has occurred. It can be performed using a variety of methods. [End of section] GAO's Mission: The Government Accountability Office, the audit, evaluation, and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO's website [hyperlink, http://www.gao.gov]. Each weekday afternoon, GAO posts on its website newly released reports, testimony, and correspondence. To have GAO e-mail you a list of newly posted products, go to [hyperlink, http://www.gao.gov] and select "E-mail Updates." Order by Phone: The price of each GAO publication reflects GAO's actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO's website, [hyperlink, http://www.gao.gov/ordering.htm]. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. Connect with GAO: Connect with GAO on facebook, flickr, twitter, and YouTube. Subscribe to our RSS Feeds or E mail Updates. Listen to our Podcasts. Visit GAO on the web at [hyperlink, http://www.gao.gov]. To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Website: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]; E-mail: fraudnet@gao.gov; Automated answering system: (800) 424-5454 or (202) 512-7470. Congressional Relations: Katherine Siggerud, Managing Director, siggerudk@gao.gov: (202) 512-4400: U.S. Government Accountability Office: 441 G Street NW, Room 7125: Washington, DC 20548. Public Affairs: Chuck Young, Managing Director, youngc1@gao.gov: (202) 512-4800: U.S. Government Accountability Office: 441 G Street NW, Room 7149: Washington, DC 20548. [End of document]