This is the accessible text file for GAO report number GAO-14-61 
entitled 'VA Vocational Rehabilitation and Employment: Further 
Performance and Workload Management Improvements Are Needed' which was 
released on January 14, 2014. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as 
part of a longer term project to improve GAO products' accessibility. 
Every attempt has been made to maintain the structural and data 
integrity of the original printed product. Accessibility features, 
such as text descriptions of tables, consecutively numbered footnotes 
placed at the end of the file, and the text of agency comment letters, 
are provided but may not exactly duplicate the presentation or format 
of the printed version. The portable document format (PDF) file is an 
exact electronic replica of the printed version. We welcome your 
feedback. Please E-mail your comments regarding the contents or 
accessibility features of this document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

United States Government Accountability Office: 
GAO: 

Report to Congressional Committees: 

January 2014: 

VA Vocational Rehabilitation and Employment: 

Further Performance and Workload Management Improvements Are Needed: 

GAO-14-61: 

GAO Highlights: 

Highlights of GAO-14-61, a report to congressional committees. 

Why GAO Did This Study: 

Veterans with disabilities face special challenges to finding 
employment—-resulting, for example, from the veterans’ physical or 
mental health, or negative attitudes or stereotypes on the part of 
some employers. VA’s VR&E program aims to help veterans with 
disabilities obtain and maintain suitable employment-—compatible with 
their disabilities—-through services such as training and job search 
assistance. The VOW to Hire Heroes Act of 2011 directed GAO to review 
the program. GAO examined (1) the outcomes for veterans seeking 
employment through the program, and (2) the progress VA has made in 
addressing critical management issues. GAO reviewed relevant laws, 
regulations, and guidance as well as recent studies; reviewed various 
program management criteria; analyzed VA administrative data on 
veterans who entered the program between fiscal years 2003 and 2012; 
interviewed staff at the VA central office and 8 regional offices; 
interviewed a random but non-generalizable sample of 17 program 
participants; and analyzed data from a VA survey of participants. 

What GAO Found: 

About half of the almost 17,000 veterans who entered the Department of 
Veterans Affairs’ (VA) Vocational Rehabilitation and Employment (VR&E) 
program in fiscal year 2003 and received employment-related services 
were placed in suitable jobs, one-third left the program, and most of 
the others are still participating. It often took veterans 6 years or 
more to achieve success (see figure), due in part to veterans often 
leaving the program temporarily. Interviews with VR&E staff and 
participants and administrative data GAO reviewed suggest veterans 
face numerous challenges that affect their ability to obtain 
employment, especially related to mental health conditions, working 
with multiple VR&E counselors over time, and civilian employers’ 
limited understanding of military work experience. 

Figure: Time to Achieve Suitable Employment for Fiscal Year 2003 
Applicants: 

[Refer to PDF for image: vertical bar graph] 

Years since application: 1; 
Percentage of veterans: 3%. 

Years since application: 2; 
Percentage of veterans: 10%. 

Years since application: 3; 
Percentage of veterans: 15%. 

Years since application: 4; 
Percentage of veterans: 17%. 

Years since application: 5; 
Percentage of veterans: 17%. 

Years since application: 6; 
Percentage of veterans: 14%. 

Years since application: 7; 
Percentage of veterans: 9%. 

Years since application: 8; 
Percentage of veterans: 7%. 

Years since application: 9; 
Percentage of veterans: 5%. 

Years since application: 10; 
Percentage of veterans: 2%. 

Source: GAO analysis of VA data. 

[End of figure] 

VA has taken steps to improve VR&E performance management, workload 
management, and staff training, but weaknesses remain. With regard to 
performance management, VA has an ongoing initiative to revise its 
approach for measuring rehabilitation success at the individual 
employee, regional, and national levels. However, the new approach VA 
is considering for employees reflects only the number, not the rate of 
successful outcomes, and therefore would not provide sufficient 
context for understanding program success. VA has not yet developed 
its new approaches for assessing rehabilitation success regionally and 
nationally. Also, VA began surveying participants’ satisfaction with 
the program and plans to use the results to manage performance; while 
VA has generally followed good survey design practices, the agency has 
not fully assessed the reliability of early customer satisfaction 
results. In terms of workload management, VA has taken steps to reduce 
paperwork burdens on regional offices. However, several offices still 
reported heavy workloads and noted that VA’s formula for allocating 
staff among offices does not consider other staff duties affecting 
workloads, such as education counseling. In addition, VA has not 
studied the relative effectiveness and efficiency of regional offices’ 
approaches for assigning staff to manage workloads. Finally, with 
respect to training, VA has addressed redundancy and most gaps in 
training for VR&E staff, but gaps remain in the areas of job placement 
assistance and workplace accommodations. 

What GAO Recommends: 

GAO recommends that VA reflect success rates in revised performance 
measures, ensure the reliability of its customer satisfaction survey 
results, re-visit its staff allocation formula, study staff 
assignments, and close certain gaps in its training for staff. In its 
comments, VA generally concurred with these recommendations and noted 
steps it plans to take to address them. 

View [hyperlink, http://www.gao.gov/products/GAO-14-61]. For more 
information, contact Daniel Bertoni (202) 512-7215 or bertonid@gao.gov. 

[End of section] 

Contents: 

Letter: 

Background: 

About Half of Veterans Seeking Employment Are Successful, Although 
Program Participants Face Many Challenges: 

VA Has Made Limited Progress in Improving VR&E Performance Management, 
Workload Management, and Training: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments and Our Evaluation: 

Appendix I: Objectives, Scope, and Methodology: 

Appendix II: Statistical Analysis of Administrative Data on Vocational 
Rehabilitation and Employment Outcomes: 

Appendix III: Vocational Rehabilitation Counselor Training Gaps 
Identified by VA in 2010 and the Agency's Efforts to Address: 

Appendix IV: Comments from the Department of Veterans Affairs: 

Appendix VGAO Contact and Staff Acknowledgments: 

Related GAO Products: 

Tables: 

Table 1: VA's Vocational Rehabilitation and Employment Performance 
Measures, with Fiscal Year 2013 Goals: 

Table 2: Description of Vocational Rehabilitation and Employment's 
Voice of the Veteran Pilot Surveys: 

Table 3: Sample Outcomes for Vocational Rehabilitation and 
Employment's Voice of the Veteran Pilot Surveys: 

Table 4: Sample Size by Fiscal Year of Program Entry and Outcomes: 

Table 5: Estimated Hazard Ratios of Time to Vocational Rehabilitation 
and Employment Program Completion: 

Figures: 

Figure 1: Total Vocational Rehabilitation and Employment Caseload and 
Counselors, and Average Caseload, Fiscal Years 2008 through 2012, and 
as of March 2013: 

Figure 2: The Vocational Rehabilitation and Employment Process from 
Application to Outcome: 

Figure 3: Outcomes to Date for Fiscal Year 2003 Applicants Who 
Received Employment-Related Services: 

Figure 4: Time to Achieve Suitable Employment for Fiscal Year 2003 
Applicants: 

Figure 5: Likelihood of Achieving Suitable Employment for Veterans 
with Selected Disabilities Compared to Those with Other Disabilities: 

Figure 6: Likelihood of Achieving Suitable Employment for Veterans 
Working with Multiple Counselors: 

Figure 7: Regional Office Caseloads for Vocational Rehabilitation 
Counselors--Lowest, Highest and Overall Average--Fiscal Years 2008 
through 2012 and March 2013: 

Figure 8: Regional Offices Contacted and Their Characteristics: 

Abbreviations: 

CWINRS: Corporate WINRS: 

EC: Employment coordinator: 

FTE: Full-time equivalent: 

Labor: Department of Labor: 

MRG: maximum rehabilitation gain: 

OMB: Office of Management and Budget: 

PTSD: Post-traumatic stress disorder: 

TBI: Traumatic brain injury: 

VA: Department of Veterans Affairs: 

VBA: Veterans Benefits Administration: 

VRC: Vocational rehabilitation counselor: 

VR&E: Vocational Rehabilitation and Employment: 

[End of section] 

GAO:
United States Government Accountability Office: 
441 G St. N.W. 
Washington, DC 20548: 

January 14, 2014: 

The Honorable Bernard Sanders: 
Chairman: 
The Honorable Richard Burr: 
Ranking Member: 
Committee on Veterans' Affairs: 
United States Senate: 

The Honorable Jeff Miller: 
Chairman: 
The Honorable Michael Michaud: 
Ranking Member: 
Committee on Veterans' Affairs: 
House of Representatives: 

About 3 million veterans--including more than 700,000 from the Iraq 
and Afghanistan era--have a disability connected to their military 
service,[Footnote 1] and many face special challenges obtaining 
employment. Veterans with disabilities may face employment challenges 
related to physical impairments, less visible mental health 
conditions, or negative attitudes and stereotypes among some 
employers. The Department of Veterans Affairs (VA) operates the 
Vocational Rehabilitation and Employment (VR&E) program to help 
veterans with service-connected disabilities obtain and maintain 
suitable employment, which it considers to be employment consistent 
with their abilities, aptitudes, and interests. The program provides 
services including vocational assessment, education and training, and 
job placement. In fiscal year 2012, about 121,000 veterans 
participated in the program at a cost of almost $1 billion. 

To improve the transition of veterans to civilian employment, Congress 
passed the VOW to Hire Heroes Act of 2011 (VOW Act).[Footnote 2] In 
addition to provisions intended to promote veterans' employment, 
including expanded access to VR&E benefits for certain unemployed 
veterans, the VOW Act mandated that GAO review the VR&E program. 
[Footnote 3] We examined (1) the outcomes for veterans seeking 
employment through the VR&E program, and (2) the progress VA has made 
in addressing critical management issues. To address these objectives, 
we reviewed relevant federal laws, regulations, and guidance as well 
as documentation related to recent management initiatives. We analyzed 
VA administrative data on VR&E participants who applied to the program 
between fiscal years 2003 and 2012, including their characteristics, 
services received, and outcomes obtained. We assessed the reliability 
of these data through electronic testing and interviews with VA staff, 
and found them sufficiently reliable for our reporting purposes. We 
interviewed VA central office staff responsible for the VR&E program, 
as well as VR&E managers and staff in 8 of VA's 57 regional offices. 
We selected offices to achieve diversity in VR&E caseload, performance 
in job placement and case processing timeliness, state unemployment 
rate, and geographic location. We conducted phone interviews with a 
random but non-generalizable sample of 17 veterans who had 
participated in the VR&E program and exited with either a successful 
or unsuccessful outcome. Finally, we obtained selected data resulting 
from a pilot VA customer satisfaction survey that includes responses 
from VR&E participants, after reviewing the survey's methodology and 
finding it to be sufficiently reliable for our purposes: 

We conducted this performance audit from October 2012 to January 2014 
in accordance with generally accepted government auditing standards. 
Those standards require that we plan and perform the audit to obtain 
sufficient, appropriate evidence to provide a reasonable basis for our 
findings and conclusions based on our audit objectives. We believe 
that the evidence obtained provides a reasonable basis for our 
findings and conclusions based on our audit objectives. Additional 
information on our scope and methodology is provided in appendix I. 

Background: 

VR&E Entitlement and Eligibility: 

To be entitled to VR&E services and related benefits, veterans 
generally must (1) have at least a 20 percent disability rating from 
the VA[Footnote 4] and (2) be in need of rehabilitation because of an 
employment handicap caused in substantial part by a service-connected 
disability.[Footnote 5],[Footnote 6] If needed to achieve employment, 
entitled veterans may receive a maximum of 48 months of vocational 
rehabilitation services and an additional 18 months of employment 
services, which include counseling, placement, and post-placement 
services.[Footnote 7] Veterans are typically eligible to receive these 
services within a 12-year period beginning on either: (1) the date of 
separation from military service, or (2) the date the veteran received 
a disability rating from VA, whichever is later. In certain 
circumstances, the 48-month limit and the 12-year period of 
eligibility may be extended.[Footnote 8] 

Availability of Additional Benefits pre- and post-VOW Act: 

Under certain circumstances, veterans who have successfully completed 
a VR&E program and achieved rehabilitation may qualify for additional 
VR&E benefits and services. Prior to the VOW Act, veterans who 
successfully completed a VR&E program could receive additional 
services if (1) their service-connected condition worsened to the 
extent that they could no longer perform the duties of the occupation 
for which they were previously rehabilitated, or (2) their occupation 
was no longer suitable because of their current capabilities and 
employment handicap.[Footnote 9] The VOW Act created another pathway 
for previously rehabilitated veterans to receive additional services. 
Under section 233 of the VOW Act, veterans who have exhausted their 
right to regular unemployment compensation under state law may receive 
up to 12 additional months of VR&E services and benefits following a 
successful rehabilitation, as long as they are otherwise eligible and 
they begin the additional program within 6 months after they exhausted 
unemployment compensation.[Footnote 10] This provision took effect on 
June 1, 2012, and veterans must apply no later than March 31, 2014 to 
qualify. 

Administrative Structure: 

Within VA's Veterans Benefits Administration's (VBA) Office of 
Economic Opportunity in Washington D.C., the central VR&E Office is 
responsible for overseeing the VR&E program. VR&E services are 
provided by managers and staff at 57 regional offices and 179 
satellite offices, under the management of VBA's central Office of 
Field Operations, which manages workload and performance to achieve 
targets set in coordination with the central VR&E Office. VR&E 
positions typically include (1) a VR&E officer who oversees the VR&E 
field office and its staff; (2) vocational rehabilitation counselors 
(VRC) who work directly with veterans to assess their entitlement, 
develop their rehabilitation plans, and manage their progress; (3) 
employment coordinators (EC) who focus on assisting veterans with job 
search and placement, which also includes employer outreach to 
facilitate contact between veterans and employers; and (4) other staff 
to support the administration of the program. VR&E field staff are 
also located at select college campuses to help veterans successfully 
complete their training and become employed, and at military sites to 
help servicemembers with disabilities as they begin their transition 
to veteran status and the civilian workforce. 

As of March 2013, 1,281 field staff were administering the VR&E 
program, of which about 70 percent (890) were VRCs. VR&E's total 
caseload has generally increased every year from 98,015 at the end of 
fiscal year 2008 to 124,100 on March 31, 2013.[Footnote 11] Over the 
same period, the number of VRCs has also generally been growing. As 
shown in figure 1, the increase in staffing has helped to bring down 
the average caseload per VRC, which grew from 137 cases to 152 cases 
between fiscal years 2008 and 2009, then steadily decreased to 139 
cases as of March 31, 2013. 

Figure 1: Total Vocational Rehabilitation and Employment Caseload and 
Counselors, and Average Caseload, Fiscal Years 2008 through 2012, and 
as of March 2013: 

[Refer to PDF for image: multiple line graph] 

FY 2008: 
Average workload per counselor: 137; 
Number of counselors: 717; 
Total workload of veteran cases: 98,015. 

FY 2009: 
Average workload per counselor: 152; 
Number of counselors: 722; 
Total workload of veteran cases: 110,050. 

FY 2010: 
Average workload per counselor: 141; 
Number of counselors: 759; 
Total workload of veteran cases: 107,073. 

FY 2011: 
Average workload per counselor: 141; 
Number of counselors: 766; 
Total workload of veteran cases: 107,925. 

FY 2012: 
Average workload per counselor: 137; 
Number of counselors: 832; 
Total workload of veteran cases: 114,281. 

March 2013: 
Average workload per counselor: 139; 
Number of counselors: 890; 
Total workload of veteran cases: 124,100. 

Source: GAO analysis of VR&E data. 

[End of figure] 

VR&E Process: 

The provision of services under the VR&E program begins with the 
veteran's application for services, a determination of entitlement and 
eligibility,[Footnote 12] and a visit to a regional or satellite 
office for further assessment.[Footnote 13] If a veteran is entitled 
and eligible to receive VR&E services and found to be employable, 
[Footnote 14] the case manager, VRC and veteran will work together to 
select one of the four VR&E employment tracks best suited for the 
veteran, to identify a suitable employment goal, and to incorporate 
that goal and the needed services and benefits to achieve it into a 
rehabilitation plan.[Footnote 15] 

Side bar: 

The four employment tracks include: 

1. Self-employment: For veterans with the skills to start their own 
business and for whom it provides the best path to employment. The 
employment plan may include, for example, development of business 
concept, management training, and guidance on obtaining funding. 

2. Re-employment: For veterans who want to return to their prior non-
military employer. The employment plan may include workplace 
accommodation and disability rights counseling. 

3. Rapid access to employment: For veterans with work skills who want 
immediate employment. The employment plan may include, for example, 
job readiness, placement, and accommodation assistance. 

4. Employment through long-term services: For veterans who need 
specialized services to obtain and maintain employment. The employment 
plan may include training, such as apprenticeship, college, or on-the-
job training; financial assistance, such as tuition assistance and 2-
month subsidy during a job search; and workplace accommodation. 

Sources: VA document and prior GAO and Congressional Research Service 
reports. 

[End of side bar] 

Once a plan is finalized, the veteran begins receiving training or 
employment services in accordance with the chosen employment track. At 
any point in this process, the veteran in consultation with the VRC 
may agree on a redirection, an additional assessment, or transition to 
another employment track or job goal. A change in medical or personal 
situation could lead the veteran to interrupt or discontinue the 
program, as well. According to VA, VR&E uses the interrupted status as 
a tool to allow veterans time to resolve unexpected issues without 
using up time-limited employment services.[Footnote 16] At the end of 
the process, one of two program closure outcomes are possible for 
veterans with a plan for employment: (1) the veteran is rehabilitated 
(i.e., successfully achieves suitable employment for 60 days or 1 year 
of self-employment) or (2) is discontinued from the program. The 
discontinuance can be with either (1) a maximum rehabilitation gain 
(MRG) (e.g., the veteran is (a) employed in a job that is not 
suitable, or (b) has chosen to not pursue any job or a suitable job, 
but planned services resulted in meaningful skills or increased 
independence which aided the veteran) or (2) without an MRG (e.g., the 
veteran has not maximized employment potential and no longer wants to 
participate in the program). Veterans who have discontinued the 
program can re-enter after VR&E has determined that they are still 
entitled to services.[Footnote 17] (See figure 2 for a summary of the 
VR&E process from application to case closure.) 

Figure 2: The Vocational Rehabilitation and Employment Process from 
Application to Outcome: 

[Refer to PDF for image: process illustration] 

Application processing: 

Veteran applies for services through the Vocational Rehabilitation and 
Employment program (VR&E): 
VR&E reviews: 
Entitled: continue; 
Not entitled: Case closed. 

Evaluation and planning: 

VR&E assesses the veteran's needs, skills, and abilities and 
determines whether the veteran can work. If needed, VR&E can also 
require that a veteran get an extended evaluation to determine the 
veteran's ability to meet the physical and mental demands of the 
workplace or a chosen vocation[A]. 
Cannot work: Independent living; 
Can work: VR&E assigns a veteran to one of four employment tracks and 
an employment plan is developed: 
1) Self-employment; 
2) Reemployment; 
3) Rapid access to employment; 
4) Employment through long term services; 
Continue. 

Employment and other services provided: 
The veteran and VR&E approve the employment plan[B]; 
Veteran pursues plan: VR&E provides agreed to services, case 
management, and other assistance as needed, such as helping the 
veteran get medical or dental services from the Veterans Health 
Administration. VR&E may also utilize job search and placement 
assistance of other public or non-profit entities, such the Department 
of Labor's Disabled Veterans’ Outreach Program. 

VR&E outcomes: 

Rehabilitated: The veteran seeking to become employed achieves and 
maintains suitable employment for 60 days or more and the veteran 
seeking self-employment maintains the business for 1 year. 

Discontinued: The veteran seeking to become employed does not maintain 
suitable employment for 60 days or more and the veteran seeking self-
employment does not maintain the business for 1 year; 
* Discontinued with maximum rehabilitation gained (MRG): Discontinued 
with MRG: VA determines that the veteran made employment-related gains 
through the program, such as becoming employable; 
* Discontinued without MRG: There are a number of reasons for why a 
veteran may be discontinued from the program[C]. 

Source: GAO analysis of VR&E documents. 

[A] If a veteran does not pursue a comprehensive evaluation or 
complete the development of a rehabilitation plan, VR&E can 
discontinue the veteran from the program without negatively affecting 
the program's performance outcomes. 

[B] If the veteran decides not to pursue the plan, VR&E may 
discontinue the veteran from the program's current rolls and will 
include the veteran's discontinuance in its performance outcomes. 

[C] Possible reasons for discontinuing the plan of a veteran who has 
not achieved an MRG from the VR&E program include the veteran: (1) has 
decided to not pursue the plan; (2) has serious medical problems that 
interfere with pursuing rehabilitation and are expected to last for an 
extended, but uncertain, period; or (3) has decided to pursue training 
through another VA education program, such as the Post-9/11 GI Bill. 

[End of figure] 

VR&E Performance Management: 

VA measures the performance of its VR&E program across several 
dimensions, and uses performance goals and data to help manage the 
program. VA provides transparency on its performance through (1) its 
annual performance and accountability report; (2) its annual budget 
submission to the Congress; and (3) a database on VA's website. 
[Footnote 18] VA's VR&E measures cover the dimensions of program 
outcomes, timeliness, and accuracy (see table 1). 

Table 1: VA's Vocational Rehabilitation and Employment Performance 
Measures, with Fiscal Year 2013 Goals: 

Outcomes: Rehabilitation rates: Percentage of veterans who were 
rehabilitated, out of veterans rehabilitated and discontinued 
without rehabilitation[A]: 

Measure: General (VA key measure); 
Description: Includes veterans in all five VR&E tracks, including the 
Independent Living track; 
FY 2013 goal: 77%. 

Measure: Employment; 
Description: Includes veterans in the four employment-related tracks; 
excludes the Independent Living track; 
FY 2013 goal: 77%. 

Measure: Serious employment handicap; 
Description:Includes veterans found by VA to have serious employment 
handicaps, in all five VR&E tracks, including the Independent Living 
track; 
FY 2013 goal: 77%. 

Measure: Independent Living; 
Description: Includes only veterans in the Independent Living track; 
FY 2013 goal: n/a[B]. 

Timeliness: Average days required to complete specific parts 
of the VR&E process: 

Measure: Entitlement determination; 
Description: Days from VR&E application to VA's notification to 
the veteran of entitlement decision; 
FY 2013 goal: 45 days. 

Measure: Evaluation and planning phase; 
Description: Days from VR&E application to (1) completion of the 
veteran's rehabilitation plan or (2) determination that the veteran is 
not entitled to VR&E services; 
FY 2013 goal: 105 days. 

Accuracy: Percentage of actions found to be accurate by VR&E 
quality reviewers: 

Measure: Entitlement determination; 
Description: Accuracy of decisions on veterans' entitlement to 
VR&E services; 
FY 2013 goal: 96%. 

Measure: Services; 
Description: Accuracy of VR&E evaluation, planning, and rehabilitative 
services provided; 
FY 2013 goal: 85%. 

Measure: Program completion; 
Description: Accuracy of determinations that participants were 
rehabilitated by, or discontinued from, the VR&E program; 
FY 2013 goal: 97%. 

Measure: Fiscal; 
Description: Accuracy of financial transactions, such as payments for 
VR&E services; for example, to educational institutions and vendors of 
goods and services for participating veterans; 
FY 2013 goal: 92%. 

Source: VA. 

Note: Includes measures VA reported in its fiscal year 2012 
Performance and Accountability Report; fiscal year 2014 Budget 
Submission; or its web-based ASPIRE Dashboard. 

[A] Does not include veterans who were not rehabilitated but obtained 
maximum rehabilitation gain from the VR&E program, or veterans who 
left the VR&E program to use Post-9/11 GI Bill benefits. 

[B] VA did not provide a fiscal year 2013 goal for this measure. Its 
fiscal year 2012 goal was 94 percent. 

[End of table] 

VA also holds the regional offices that administer the VR&E program 
accountable through performance goals, both for regional VR&E 
management and for individual VRC and EC staff members. To accomplish 
this, VA uses the above goals, as well as additional measures and 
goals. For example, individual VRCs are evaluated on case production 
and participants' outcomes based on the general and serious employment 
handicap rehabilitation rates; the number of successful 
rehabilitations they complete; and the percentage of their caseloads 
in interrupted status. 

About Half of Veterans Seeking Employment Are Successful, Although 
Program Participants Face Many Challenges: 

About Half of Veterans Find Suitable Jobs, but Time Frames Can Be 
Lengthy: 

About half of the veterans who entered the VR&E program in fiscal year 
2003 and received employment-related services have obtained suitable 
jobs.[Footnote 19] Of the 45,520 veterans who applied in fiscal year 
2003, 16,726 received some type of employment-related services. 
[Footnote 20] Among veterans in this group, 7,686 had achieved the 
program goal of suitable employment by the end of fiscal year 2012. 
Many others--4,841--had discontinued (stopped participating) and not 
yet re-applied. Although these veterans have not achieved suitable 
employment, a substantial number are considered by VA to have attained 
MRG and therefore some benefits from participating. In fact, according 
to VA data, some of these veterans were ultimately placed in a job, 
but the job was not seen as suitable given their disability. Most of 
the remaining veterans who started in fiscal year 2003 and received 
employment-related services were still participating in VR&E services, 
while a few had transferred to the independent living program within 
VR&E and achieved rehabilitation.[Footnote 21] (See figure 3.) 

Figure 3: Outcomes to Date for Fiscal Year 2003 Applicants Who 
Received Employment-Related Services: 

[Refer to PDF for image: pie chart] 

Placed in suitable job: 47%; 
Still participating: 20%; 
Achieved independent living goal: 2%; 
Discontinued: 30%: of that 30%: 
* Maximum rehabilitation gain (MRG): 30%; 
* Other reason: 70%. 

Source: GAO analysis of VA data. 

Note: The percentages in the four outcome categories do not add up to 
100 percent due to rounding. It is possible that some veterans who 
discontinued in fiscal year 2010 or later are included as MRG cases in 
this figure even though they actually discontinued for a different 
reason. This is because of a change in fiscal year 2010 in the meaning 
assigned to certain discontinuation reason codes in VA data. 

[End of figure] 

While the amount of time veterans spend in the VR&E program varies 
substantially among participants, it can be lengthy for many veterans. 
Among those who started in fiscal year 2003, the average time from 
application to successful placement in a suitable job was 4 years 7 
months. While about half of these veterans completed the program 
within 3 to 6 years, many took considerably longer (see figure 4). 
Furthermore, it appears that time frames for completing the program 
have not changed substantially for more recent applicants. For 
example, looking at VR&E participants who successfully attained 
suitable employment during fiscal year 2012, we found their average 
time from application to completion was also 4 years 7 months, with a 
range similar to that for fiscal year 2003 applicants. Meanwhile, 
among veterans who entered in fiscal year 2003 and have discontinued 
without yet returning, the average time from application to the final 
discontinuation was very similar--5 years. 

Figure 4: Time to Achieve Suitable Employment for Fiscal Year 2003 
Applicants: 

[Refer to PDF for image: combined vertical bar and line graph] 

Years since application: 1; 
Percentage (and number) of veterans[A]: 3% (214);
Cumulative percentage of veterans[B]: 3%. 

Years since application: 2; 
Percentage (and number) of veterans[A]: 10% (749); 
Cumulative percentage of veterans[B]: 13%. 

Years since application: 3; 
Percentage (and number) of veterans[A]: 15% (1,138); 
Cumulative percentage of veterans[B]: 27%. 

Years since application: 4; 
Percentage (and number) of veterans[A]: 17% (1,290); 
Cumulative percentage of veterans[B]: 44%. 

Years since application: 5; 
Percentage (and number) of veterans[A]: 17% (1,318); 
Cumulative percentage of veterans[B]: 61%. 

Years since application: 6; 
Percentage (and number) of veterans[A]: 14% (1,094); 
Cumulative percentage of veterans[B]: 76%. 

Years since application: 7; 
Percentage (and number) of veterans[A]: 9% (730); 
Cumulative percentage of veterans[B]: 85%. 

Years since application: 8; 
Percentage (and number) of veterans[A]: 7% (557); 
Cumulative percentage of veterans[B]: 92%. 

Years since application: 9; 
Percentage (and number) of veterans[A]: 5% (409); 
Cumulative percentage of veterans[B]: 98%. 

Years since application: 10; 
Percentage (and number) of veterans[A]: 2% (187); 
Cumulative percentage of veterans[B]: 100%. 

Source: GAO analysis of VA data. 

Note: Percentages of veterans placed in each year do not add up to 100 
due to rounding. 

[A] Indicates the percentage of successfully rehabilitated fiscal year 
2003 applicants placed in suitable employment in each year following 
application. 

[B] Indicates the cumulative percentage of successfully rehabilitated 
fiscal year 2003 applicants placed in suitable employment by each year 
following application. 

[End of figure] 

Factors contributing to varied but generally lengthy time frames 
include the services veterans receive, as well as the fact that many 
extend their overall time in the program by interrupting or 
discontinuing along the way. Of those veterans who started since 
fiscal year 2003 and received some employment services, most (about 70 
percent) pursued a 4-year college degree, and these veterans typically 
took longer to complete the program than those who pursued other types 
of training such as community college degrees or vocational training. 
In addition, we estimate that among veterans who started since fiscal 
year 2003 and achieved suitable employment, about one-quarter 
interrupted their programs at least once.[Footnote 22] Almost the same 
proportion discontinued and returned to the program at least once 
before ultimately achieving success, and some did so two or three 
times. The small proportion (4 percent) who received job placement 
assistance only, meanwhile, attained suitable employment more quickly 
than those who received other types of services. 

Once veterans attain suitable employment, they rarely come back for 
additional services, even though they may be entitled to such services 
if, for example, their disability worsens or they qualify under new 
expanded criteria established by the VOW Act. Of veterans who entered 
the program in fiscal year 2003 and were placed in suitable jobs, 4 
percent have applied for and 2 percent have actually received 
additional employment-related VR&E services, with 3 years on average 
between completing the program and re-applying. The data on applying 
again following rehabilitation are similar for veterans who entered in 
fiscal years 2004 and 2005. Moreover, the expanded eligibility 
criteria authorized by the VOW Act have not so far had a substantial 
impact on the number of veterans returning for additional services. 
According to data provided by VA, between May 2012 and April 2013 only 
10 veterans nationwide received additional VR&E services under these 
new VOW Act criteria. Managers or staff in a number of regional 
offices told us the new criteria would not significantly expand the 
population of veterans eligible for additional VR&E services: several 
noted that almost all the veterans who would qualify under the new 
criteria are already eligible under the old criteria. Some also said 
it is preferable to obtain additional services under the old criteria, 
which provide more months of additional services.[Footnote 23] 

A Variety of Challenges May Result in Many Veterans Not Completing the 
Program: 

While veterans may discontinue for a variety of reasons, analysis of 
administrative data, results from a VA customer satisfaction survey, 
and our interviews with VR&E staff and participants point to several 
key challenges that may affect veterans at any stage of the 
rehabilitation process.[Footnote 24] 

Mental Health and Other Disabilities: 

Veterans' disabilities--especially those related to mental health--
present challenges throughout the rehabilitation process. Among 
respondents to VA's Voice of the Veteran benchmark survey who had 
withdrawn from or interrupted their rehabilitation programs, "medical 
difficulties" was the most frequently cited reason for doing so (about 
one-third cited this as a reason).[Footnote 25] Further, our analysis 
of VA administrative data found that veterans with mental health 
conditions were 12 percentage points less likely to be successful 
within 8 years of program entry than those without such conditions, 
the largest difference for any specific type of disability (see figure 
5).[Footnote 26] VR&E managers, staff, and program participants we 
interviewed provided examples of how medical conditions can be an 
obstacle. For example, a VRC in one office explained that veterans who 
stop participating in a training program when they undergo surgery for 
their service-connected disability may lack motivation to resume 
training afterwards. A VRC in another office said veterans with post 
traumatic stress disorder (PTSD) face special challenges with 
maintaining employment because of difficulties interacting with co-
workers. One veteran we interviewed said he completed a certificate 
program to become a heavy equipment operator, but was unable to obtain 
the required class D driver's license because his PTSD made it 
difficult for him to deal with the noise and crowds at the testing 
site. Another veteran told us he trained for and was placed in a job 
in information technology. However, he said his job involves tasks for 
which he had not been trained, and his mental health conditions--
including traumatic brain injury (TBI)--have made it hard for him to 
learn these tasks and may force him to leave his job. 

Figure 5: Likelihood of Achieving Suitable Employment for Veterans 
with Selected Disabilities Compared to Those with Other Disabilities: 

[Refer to PDF for image: vertical bar graph] 

Percentage difference compared to veterans without selected disability: 

Mental health: -12%; 
Immunological: -5%; 
Neurological: -2%; 
Cardiovascular: 2%; 
Skin: 4%; 
Respiratory: 5%; 
Gynecological: 5%; 
Ear: 5%. 

Source: GAO analysis of VA data. 

Note: This figure presents estimates of the difference in the 
likelihood of success within 8 years for veterans with each disability 
compared to those without the disability. The figure does not include 
results for 6 other disabilities for which we did not find a 
statistically significant association with the likelihood of success, 
after controlling for other factors. 

[End of figure] 

Developing a Realistic Employment Plan: 

VR&E staff and veterans often find it challenging to develop 
employment plans that can realistically lead to success. 
Administrative data suggest there are difficulties with developing a 
sound plan. About half the veterans who started since fiscal year 2003 
and received some employment-related services have stopped their 
programs and returned for additional evaluation and planning at least 
once, and some have done so multiple times.[Footnote 27] Managers or 
staff in all eight offices we contacted said some veterans enter the 
program with unrealistic employment goals, which makes vocational 
planning more challenging. For example, VR&E staff in one office said 
veterans may want to train for jobs in law enforcement, even though 
they have PTSD or another psychological condition that makes this goal 
unrealistic. Some veterans we interviewed cited challenges coming to 
agreement with their counselors on an employment plan. For example, 
one said he wanted to study theology, but his counselor felt that 
studying to become a teacher would be more appropriate, and he 
ultimately left the program because of this disagreement. In addition, 
managers or staff in all eight offices we contacted said veterans may 
come to the VR&E program with the goal of furthering their education 
rather than obtaining a job, or may have a misperception about the 
purpose of the VR&E program. For example, a VRC in one office said she 
must explain to veterans that VR&E only provides training when it is 
necessary to achieve employment. She feels veterans' chance of success 
would be enhanced if they entered the program with specific employment 
goals in mind, rather than training goals. Input from veterans is 
consistent with this viewpoint. While most recently-enrolled 
respondents to the Voice of the Veteran survey said they were 
motivated to apply to the VR&E program by employment-related goals, a 
substantial number (about a quarter) said they applied primarily to 
further their education. 

Multiple VR&E Counselors: 

Analysis of administrative data indicates that the more VR&E 
counselors a veteran works with during his or her participation in the 
program, the less likely he or she is to achieve success. As noted 
above, veterans typically spend several years participating in the 
program, and most work with multiple counselors. The vast majority of 
those who entered the program since fiscal year 2003 and received some 
employment services have worked with at least two counselors, and a 
substantial proportion (about one-third) worked with four or more. We 
found that the chance of success declines steadily as veterans work 
with more counselors (see figure 6).[Footnote 28] For example, 
veterans who had worked with two counselors were 8 percentage points 
less likely to have attained suitable employment within 8 years of 
entering the program, as compared to those who had worked with one 
counselor. Veterans who had worked with four counselors were 27 
percentage points less likely to have attained suitable employment. 

Figure 6: Likelihood of Achieving Suitable Employment for Veterans 
Working with Multiple Counselors: 

[Refer to PDF for image: vertical bar graph] 

Number of counselors: 2; 
Percentage difference compared to veterans with one counselor: -8%. 

Number of counselors: 3; 
Percentage difference compared to veterans with one counselor: -17%. 

Number of counselors: 4; 
Percentage difference compared to veterans with one counselor: -27%. 

Number of counselors: 5; 
Percentage difference compared to veterans with one counselor: -39%. 

Number of counselors: 6; 
Percentage difference compared to veterans with one counselor: -51%. 

Number of counselors: 7; 
Percentage difference compared to veterans with one counselor: -60%. 

Number of counselors: 8; 
Percentage difference compared to veterans with one counselor: -70%. 

Number of counselors: 9+; 
Percentage difference compared to veterans with one counselor: -71%. 

Source: GAO analysis of VA data. 

Note: This figure presents estimates of the percentage difference in 
the likelihood of success within 8 years for veterans who worked with 
more than 1 counselor compared to those who worked with only 1 
counselor. 

[End of figure] 

Family and Financial Obligations: 

According to the Voice of the Veteran survey, many veterans who 
withdrew from or interrupted their employment services cited "family 
obligations" (27 percent) and "financial difficulties" (18 percent) as 
a reason. Managers or staff in seven of the eight regional offices we 
contacted also said one or both of these are common issues for 
veterans. For example, according to managers in one office, veterans 
may have an immediate need to earn more money to support their 
families, so they quit training and take a job that may not be 
suitable given their disability. VRCs in another office told us a 
change in life circumstances--such as marriage, the birth of a child, 
or a death in the family--may make it more difficult to complete 
training. Veterans we interviewed also mentioned family obligations as 
a challenge. For example, one said she failed to complete her 
certification program in school administration because of the birth of 
her child. 

Transition from Military to Civilian Employment: 

Managers, staff, and veterans we interviewed cited challenges 
associated with the transition from military service to civilian 
employment. For example, managers or staff in five of the eight 
regional offices we contacted told us veterans may have difficulty 
translating military experience into terms that civilian employers can 
understand, or that they may need civilian credentials to get a job 
even with the expertise they gained in the military. A VRC in one 
office explained that, in the state served by his office, veterans who 
received technical training in the medical field while in the military 
may still need additional training to pass a state certification exam 
to work in a comparable civilian job. Along similar lines, one veteran 
said he had responsibilities for benefits counseling, training, and 
recruitment while in the military that are comparable to the skills 
needed in a civilian human resources job. However, when he sought 
employment with the federal government, his VR&E counselor could not 
help him translate his military experience into federal civilian 
employment. In addition, staff in three of the offices we contacted 
said civilian employers may not understand veterans' disabilities and 
how to accommodate them. One veteran we interviewed told us he was 
struggling to perform his job duties because of his physical 
disabilities, but had not received needed accommodations from his 
employer. He also felt his VR&E counselor did not sufficiently explain 
the physical demands of the job in advance. Finally, staff in two 
offices said veterans may lack experience with civilian work culture, 
which can present difficulties with succeeding in a new job. For 
example, an EC in one office said veterans are accustomed to having 
standard operating procedures in the military, and may struggle in a 
civilian job that lacks such clear guidelines. 

Regional Office Variation Due to Economic or Other Factors: 

We found that veterans served by some regional offices were 
significantly less likely to succeed than those served by others, 
after controlling for a variety of other factors.[Footnote 29] 
However, we were unable to control for local economic conditions in 
our analyses,[Footnote 30] and managers or staff in 4 of 8 offices we 
contacted said the local labor market can be a key challenge to 
placing veterans in suitable jobs. VA central office officials also 
said variations in local economic conditions, including variation in 
the unemployment rate and the availability of good-paying jobs, 
contribute to regional variation. For example, they cited the loss of 
manufacturing jobs in the area served by one office as part of the 
explanation for its lower than average success rate. VA officials also 
cited other potential factors in regional variation, such as offices' 
ability to retain experienced staff and fluctuations in staffing 
levels due to retirements. For example, they said the VR&E workforce 
in one office with below average performance may be more transient 
than in other offices. 

VA Has Made Limited Progress in Improving VR&E Performance Management, 
Workload Management, and Training: 

VA Faces Challenges with Measuring Employment Outcomes and Veteran 
Satisfaction: 

VR&E Outcome Measurement Presents a Challenge for VA: 

VA has identified limitations with its primary measure of VR&E success-
-the rehabilitation rate--and is considering replacing it with a 
numeric target for positive outcomes. VA central and regional office 
officials noted that the rehabilitation rate goal can incentivize a 
VRC to hold onto an inactive case, even if the veteran has clearly 
left the program, because holding onto inactive cases instead of 
closing them can help regional offices and individual VRCs increase 
their rehabilitation rates and more easily reach their goals for a 
particular fiscal year.[Footnote 31] When staff delay closing cases 
like this, an office's or individual's rehabilitation rate may appear 
higher than it really is in one year and lower in another year, 
leading to an inaccurate picture of VA's success in rehabilitating 
veterans. Furthermore, while cases remain in interrupted status, VRCs 
are still required to attempt contact with veterans every 60 days, 
which can limit the time spent working with veterans who are actively 
participating. In addition to the incentive to hold onto inactive 
cases, VA officials noted that the rehabilitation rate does not 
capture all of the benefits VR&E provides to veterans. In particular, 
the current calculation excludes MRG cases, where the veteran obtained 
some value from VR&E services but did not obtain and maintain suitable 
employment.[Footnote 32] Because it does not explicitly include MRG 
cases in the assessment of VR&E performance, VA's current measure does 
not fully reflect the value it provides to veterans. Another concern 
raised by some regional officials was that having a uniform 
rehabilitation rate goal for every office does not account for 
regional differences in factors affecting employment, such as local 
job market conditions.[Footnote 33] 

To address these and other issues, VA established four workgroups to 
review and make recommendations for improving VR&E's individual 
employee performance standards.[Footnote 34] The resulting draft 
recommendations included eliminating VR&E's rehabilitation rate 
measure and replacing it with a measure of the number of positive 
outcomes--specifically, the number of successful employment and 
independent living rehabilitations and MRG cases.[Footnote 35] Four 
regional offices were recommended to pilot the revised measure. 
According to a VA official, VA is expected to approve these draft 
recommendations by the end of fiscal year 2014. Also, in comments 
provided after reviewing a draft of this report, VA stated that it is 
developing VA-wide and regional office measures of program success, 
and expects to implement the VA-wide measures in fiscal year 2015. 

While the current rehabilitation rate has limitations, the measure VA 
is considering as an alternative may not necessarily be a better way 
to monitor and demonstrate VR&E's success. Specifically, VA is 
considering replacing its current outcome measure--which tracks those 
who obtained suitable jobs relative to those who did not--with a 
measure that tracks only the number of any positive outcomes. 
According to VA officials, positive outcomes would include not just 
veterans who obtained suitable employment, but also veterans who 
obtained other benefits (such as MRG) from the VR&E program. Good 
practices for performance measurement suggest that measures should 
allow agencies to transparently report their success to the Congress 
and public, and identify and address performance variances across 
their programs.[Footnote 36] The proposed measure of specific numbers 
of positive outcomes lacks this transparency because it does not 
account for veterans who left the program without obtaining suitable 
employment or other discernible benefits. It also obscures VA's 
success in achieving its core mission because it does not separate 
veterans who obtained suitable employment from those who received 
other, lesser benefits. Finally, measuring only numbers of positive 
outcomes is less effective for monitoring and managing performance 
across regional offices--that is, comparing each office's relative 
rate of success to identify those that may serve as models or require 
further oversight. 

Aside from considering changes to its outcome measures, VA has taken 
other steps to help regional offices better manage their interrupted 
cases, but it is unclear how successful these efforts have been. For 
example, VA uses performance goals to hold regional offices and VRCs 
accountable for monitoring interrupted cases and keeping their 
proportions below certain thresholds. The current goal is 11 percent 
for regional offices and 12 percent for individual VRCs. Also, VR&E's 
central office has provided tools and training to help VR&E managers 
monitor the number and average time of cases in interrupted status, 
and percent in interrupted status over 180 days. VR&E managers at two 
regional offices we contacted noted that they had VRCs close old, 
inactive cases, regardless of the short-term impact on the office's 
rehabilitation rate. VA reported that the overall percentage of 
interrupted cases declined from 10.4 percent in fiscal year 2010 to 
9.6 percent in fiscal year 2012. However, managing interrupted cases 
remains a challenge. According to VA data, in July 2013, 39 percent of 
more than 15,000 interrupted cases had been in that status for more 
than 180 days. In addition, 8 of 57 regional offices had at least 50 
percent of their interrupted cases in that status for more than 180 
days, ranging up to 70 percent at one office. 

VA Lacks Performance Measures of Veterans' Long-term Employment 
Success: 

The mission of the VR&E program includes helping veterans maintain 
suitable employment, and VA holds itself accountable for achieving 
this part of the mission by counting job placements as rehabilitations 
if they are maintained for at least 60 days. According to several 
regional office managers and staff we interviewed, cases are typically 
closed when veterans have maintained employment for this 60-day 
minimum, as problems with a veteran maintaining employment were most 
likely to occur in the first 60 days of employment if they occurred at 
all. However, if a veteran appears to be at high risk of losing a job, 
for example, the VRC can keep the case open for a longer period to 
provide additional supports. Some VRCs noted that this can occur when 
a veteran has a serious mental health condition, for example.[Footnote 
37] According to VA data on about 38,000 veterans who applied for VR&E 
between fiscal years 2003 and 2012 and have been rehabilitated to 
employment, over one-third were tracked in employment for 6 months or 
longer before having their cases closed as rehabilitated, and some 
were tracked for 1 year or more.[Footnote 38] A VA official noted that 
some of this time is spent completing documentation and approval of 
case closures. In addition, one regional VR&E manager noted that 
closure may be delayed when the veteran does not inform the regional 
office that he or she obtained a job. 

Research and practices at other federal employment programs suggest 
that tracking employment for 60 days may not be sufficient to gauge 
whether a veteran will be able to maintain employment. For example, a 
2013 National Organization on Disability study suggested that wounded 
warriors transitioning to civilian employment need support for an 
extended time after obtaining jobs. Meanwhile, the Department of Labor 
(Labor) measures individuals' employment retention over a longer term, 
180 days.[Footnote 39] This measure applies to many of Labor's 
employment programs, including its veterans employment programs. 

VA officials stated that they plan to collect information on veterans' 
long-term employment, but did not indicate whether they plan to 
develop performance measures to show the program's impact on veterans' 
long-term employment. Specifically, VA is considering contacting 
rehabilitated and discontinued VR&E participants at 6 and 12 months 
after they finish the program. This effort could yield information on 
employment over the longer term. VA plans to use this information to 
identify veterans who are having difficulties in maintaining 
employment, and possibly provide them additional supports. However, 
without developing performance measures, VA may be missing an 
opportunity to track the long-term effectiveness of the VR&E program. 

VA Generally Followed Recognized Practices in Designing Its Customer 
Satisfaction Survey, but the Quality of Benchmark Results is Uncertain: 

VA has made progress toward holding itself accountable for VR&E 
participants' satisfaction by implementing its new Voice of the 
Veteran survey, but the agency has not incorporated practices 
necessary to ensure the quality of benchmark results. VA intends to 
use survey data to inform program improvements and training 
developments, and potentially to report on veterans' satisfaction with 
VR&E.[Footnote 40] In developing and piloting the survey,[Footnote 41] 
VA followed generally recognized survey practices, including detailed 
survey planning, comprehensive questionnaire development and testing, 
and extensive data collection activities, such as follow-up with 
veterans who did not respond to VR&E's initial survey request. 
[Footnote 42] However, VA did not perform a nonresponse analysis to 
determine the effects of any nonresponse bias on the quality of its 
pilot survey data.[Footnote 43] Guidance from the Office of Management 
and Budget (OMB) states that agencies should perform nonresponse 
analysis when a survey's unit response rate is below 80 percent, to 
identify the possibility of bias in a survey's results, and measure, 
adjust for, report, and analyze nonresponse to assess its effects on 
data quality.[Footnote 44] VA's plan for the pilot survey, as approved 
by the OMB, anticipated the need for this analysis, but its contract 
with the survey administrator made it optional.[Footnote 45] Although 
the response rates for the three pilot surveys of VR&E participants 
each fell below the 30 percent level specified in VA's sample design, 
VA officials said they elected not to conduct a nonresponse analysis 
due to budget constraints.[Footnote 46] Moving forward, VA officials 
stated there is no expectation that a nonresponse analysis be 
performed for VA's ongoing survey effort unless deemed necessary. 
Currently, VA's approved contract for its ongoing survey effort does 
not include an option for a nonresponse analysis if response rates are 
low. 

VA Workload Management Efforts May Not Fully Address Challenges Facing 
Some Regional Offices: 

VA has acted to strengthen VR&E workload management by commissioning a 
study of workload management issues, among other things. The study 
team collected and analyzed data on how VR&E staff at four regional 
offices spent their time on a variety of tasks, such as paperwork and 
client contacts. In its April 2011 report,[Footnote 47] the study team 
recommended, among other things, that VA improve its approach for 
estimating its overall staffing needs. VA now has an ongoing 
initiative to assess the total number of VR&E staff needed across the 
regional offices. This initiative, intended for use in formulating 
future VR&E administrative budget requests, is expected to be 
completed by March 2014. Also, in response to an internal business 
process review, VA has made changes to VR&E procedures intended to 
ease VRCs' workloads by reducing paperwork. 

Despite these efforts, some regional offices struggle with high 
caseloads that may affect quality of services to veterans, while 
others do not, suggesting potential shortcomings in the way VA 
allocates resources. VR&E managers and staff in four of the offices we 
contacted said high workloads impede their ability to spend sufficient 
time working with individual veterans. Meanwhile, VRCs in other 
offices we contacted said their workloads were manageable. VA 
administrative data indicate that average ratios of VR&E cases to VRCs 
vary considerably, as shown in figure 7. For example, while VA 
reported a national average of 139 cases per VRC as of March 31, 2013, 
regional office average caseloads ranged from 64 in Fargo, North 
Dakota to 579 in White River Junction, Vermont.[Footnote 48] As of 
March 31, 2013, seven offices averaged fewer than 100 cases per VRC, 
[Footnote 49] while eight averaged more than 175 cases per VRC. 
[Footnote 50] 

Figure 7: Regional Office Caseloads for Vocational Rehabilitation 
Counselors--Lowest, Highest and Overall Average--Fiscal Years 2008 
through 2012 and March 2013: 

[Refer to PDF for image: multiple line graph] 

FY 2008: 
Highest office: 188.7; 
VR&E-wide average: 136.7; 
Lowest office: 53. 

FY 2009: 
Highest office: 279.2; 
VR&E-wide average: 152.4; 
Lowest office: 50.5. 

FY 2010: 
Highest office: 327.1; 
VR&E-wide average: 141.1; 
Lowest office: 62.5. 

FY 2011: 
Highest office: 276; 
VR&E-wide average: 141; 
Lowest office: 89.8. 

FY 2012: 
Highest office: 542; 
VR&E-wide average: 139.5; 
Lowest office: 67. 

March 2013: 
Highest office: 579; 
VR&E-wide average: 139.4; 
Lowest office: 64.3. 

Source: GAO analysis of VR&E workload reports. 

[End of figure] 

VA officials cited factors that can contribute to the wide variations 
in caseload ratios across regional offices, such as different case 
management approaches or VRC attrition than can result in temporarily 
high caseloads. Further, some offices place VRCs at military treatment 
facilities and college campuses. VA officials noted that these VRCs--
because of the different service needs of servicemembers and veterans 
at these locations[Footnote 51]--may have different caseload levels. 

According to VA, the resource allocation model VA uses to set the full-
time equivalent (FTE) employee level in each office and assign any new 
FTE positions each year accounts for three factors that may affect 
staffing needs.[Footnote 52] For fiscal year 2013, the model was based 
on workload (50 percent), outcome performance (25 percent), and 
accuracy performance (25 percent).[Footnote 53] However, VR&E managers 
in five of the eight regional offices we contacted identified gaps and 
issues with VA's criteria for allocating VR&E staff across offices 
that could limit the model's effectiveness in assigning resources 
where they are most needed. Issues cited by these officials included: 

* The model does not consider workloads associated with educational 
and vocational counseling cases,[Footnote 54] which may be 
disproportionately high in certain offices, particularly where there 
is a large military presence. For example, according to VA data, the 
San Diego office had over one-third of all educational and vocational 
counseling cases in fiscal year 2012. Regional offices expend staff 
resources on these cases, but do not receive credit for them in the 
resource allocation model. 

* The model's use of performance measures may have the effect of 
taking staff away from poorer-performing offices, while increasing 
staffing at better-performing offices. This could make it harder for 
poorer-performing offices to improve their performance.[Footnote 55] 

According to leading practices for developing resource allocation 
models, agencies should ensure the credibility of the data used in 
such models. For example, the data should reflect correct assumptions 
about how staff time is actually spent. Also, agencies should consider 
ideas and information from people who work at the affected offices and 
other stakeholders when designing the model.[Footnote 56] For VR&E, 
this could include VR&E staff at regional offices. VA officials stated 
that they have not made significant revisions to the resource 
allocation model since fiscal year 2003, but have made adjustments, 
such as adding the accuracy factor. However, VA officials noted that 
they plan to re-assess the VR&E resource allocation model after they 
have developed their new model for estimating overall VR&E staffing 
needs. 

In addition to issues with its resource allocation model, VA lacks 
information about regional offices' varied approaches to managing 
their caseloads. VA's administration of the VR&E program is 
decentralized; according to VA central office officials, each regional 
office's management is free to decide how best to manage its office's 
workloads. Thus, regional offices have adopted a variety of approaches 
to managing their VR&E workloads. Some offices generally assign a 
veteran to work with one VRC throughout the rehabilitation process, 
until the point at which the veteran is determined to be job ready. 
Meanwhile, other regional offices have VRCs specialize in one phase of 
the rehabilitation process, so a veteran works with multiple VRCs as 
he or she progresses through the program. For example, some VRCs may 
deal with just the front end of the process--determining veterans' 
entitlement to services and preparing rehabilitation plans--while 
others focus on managing cases of veterans who are pursuing their 
rehabilitation plans.[Footnote 57] Also, some regional offices have 
VRCs who specialize in certain types of cases, such as veterans in the 
independent living track. Another variation at some offices is to 
assign a VRC to veterans attending a specific college or other 
education and training institution. 

Officials at the regional offices we contacted had different opinions 
about the efficacy of VRCs specializing by phase of the VR&E process. 
Some VR&E managers and staff cited advantages to allowing a single VRC 
to handle a case: for example, a VRC can (1) develop a stronger 
rapport with the veteran; and (2) avoid possible confusion, including 
among the veterans, related to handoffs between VRCs. Other regional 
officials cited advantages of specialization. For example, the VRC 
can: (1) manage caseloads without being distracted by a larger range 
of tasks (for example, assessing entitlement, preparing rehabilitation 
plans and monitoring veterans' progress on their plans); and (2) 
develop expertise in one phase of the VR&E process. VR&E managers at 
one of the regional offices we contacted stated that they plan to move 
away from a more specialized caseload management approach, citing 
improved efficiency from having one VRC handle a case. Meanwhile, 
managers in another office said they had recently moved towards a 
specialization approach because they feel it better enables them to 
handle their large caseload. 

Federal internal control standards require that managers obtain 
operational data to ensure their programs are run effectively and 
efficiently.[Footnote 58] For VA, this could include data on how staff 
resources are being utilized across regional offices, and how such 
usage affects offices' efficiency and effectiveness. It is especially 
important that VA understand regional offices' approaches and their 
effects, given our finding that veterans who work with a number of 
different VRCs over time are less likely to be successful. Moreover, 
the April 2011 consultants' report to VA on VR&E workload management--
which also found wide variations in how VRCs at different offices 
allocated their time among different tasks--recommended that VA assess 
regional office approaches to assigning tasks among VRCs in order to 
identify best practices for helping to rehabilitate veterans, while 
balancing standardization across offices with flexibility to serve 
each office's service area. However, VA central office does not 
collect information on how regional offices manage their caseloads and 
assign tasks among VRCs, and has not assessed the effect of varied 
regional office approaches on case processing efficiency and veterans' 
rehabilitation success. VR&E central office officials stated that they 
prefer to leave workload management to regional office managers 
because they are in the best position to determine which workload 
management approach will enable the office to meet its program goals. 

VA Is Addressing Redundancy and Most Gaps in VR&E Staff Training: 

In the course of our work, we found evidence that staff had 
experienced redundancy and critical gaps in their training; however, 
VA had taken recent steps to address the redundancy and most, but not 
all of the training gaps. Principles of effective strategic training 
and workforce development call for avoiding duplication, identifying 
workforce competency gaps, and addressing these gaps.[Footnote 59] 
Nonetheless, some of the VR&E regional managers or staff at six of the 
eight offices we contacted told us that the annual curriculum offered 
to VRCs and ECs required them to repeat courses. Their experience 
notwithstanding, we found that VA has recently been adding new courses 
and, by fiscal year 2013, none of the courses on the curricula for 
experienced VRCs and ECs were redundant with those offered in the 
prior year. Further, with the addition of these new courses, VA is 
actively closing the nine critical training gaps identified in a June 
2010 skill gap analysis report.[Footnote 60] The report noted that 
five gaps affect both new and experienced VRCs and four additional 
gaps affect only new staff. VBA contracted for this analysis in 
response to a report we issued in 2009, in which we found that VR&E 
had not collected data on the critical skills and competencies needed 
by VRCs to ensure they could meet the current and future needs of 
veterans in the program.[Footnote 61] In September 2013, VA informed 
us that it had begun providing training to staff to address these gaps 
and will have provided training to address all the gaps by the end of 
2013, and in 2014 will provide two new courses for staff on mental 
health to improve their ability to assist veterans with PTSD and other 
mental illnesses. See appendix III for the gaps identified and 
training developed to address them. 

Although VR&E closed many training gaps, our review of VR&E's fiscal 
year 2013 curriculum identified other gaps related to job placement 
and accommodation for experienced VRCs. Staff or managers in five of 
the eight regional VR&E offices contacted also identified either job 
placement or workplace accommodation as areas in which additional 
training was needed. For example: 

* Some regional managers and staff told us VRCs need more training in 
job placement strategies, employment laws, and tax incentives for 
employers.[Footnote 62] Job placement skills are critical to achieving 
VR&E's main mission to ensure that veterans with disabilities who can 
be employed obtain and maintain suitable employment. While ECs are 
generally responsible for assisting veterans with job placement, VRCs 
may take on this task in offices that lack ECs, or when ECs simply are 
not able to provide these services to veterans needing them. 

* Some regional managers and staff told us VRCs needed training in 
current workplace accommodations to better assist veterans. Workplace 
accommodations help ensure veterans with disabilities can successfully 
perform on the job and maintain their employment. A June 2012 VR&E-
sponsored study confirmed the importance of workplace accommodations 
to help veterans attain and maintain long-term employment and 
recommended that VR&E either train or retrain its staff in workplace 
accommodation.[Footnote 63] 

Although VA does not currently have specific plans to address either 
of these gaps in fiscal year 2014, VA officials informed us that the 
agency intends in the future to individually identify and address 
skill gaps for each VRC and EC. In the meantime, regarding job 
placement training, VA officials told us that VRCs may opt to take 
online training developed for its ECs, but that VA does not plan to 
require VRCs to take job placement training because VRCs receive it as 
part of their college or master's degree programs. However, VRCs may 
not opt to take optional VA training as several VRCs and managers told 
us that VRCs lack time to take all the required training. As of July 
2013, VA officials had not yet developed the curriculum for fiscal 
year 2014 and did not know if training on workplace accommodation for 
VRCs would be included. 

As for VA's plans to identify and address each VRC's and EC's skill 
gaps, this effort is part of its strategic initiative to provide 
competency-based training for 16 critical positions throughout the 
agency. VA officials informed us that under this initiative, which 
began in 2010, the agency plans to design and provide tailored 
training and other performance support that will be based on an 
assessment of each critical employee's current proficiency level. VR&E 
officials expect the outcomes of this effort to include: (1) 
individual performance standards, (2) tailored performance support, 
(3) an assessment baseline and developmental roadmap (i.e., training 
plan), and (4) a tailored National Training Curriculum and remediation 
plan for each VRC and EC. Furthermore, through the collection and 
management of competency data, VR&E officials hope to track the 
knowledge, skills, abilities, education, and experience required for 
specific positions and recruit, train, distribute, and professionally 
develop personnel based on this competency framework. VA officials 
reported that as of fiscal year 2012, 47 percent of employees in 
critical and key occupations had participated in a competency-based 
training program within the last 12 months, and that in June 2013 the 
agency took an initial step to provide individualized training for 
VRCs. The program and plans VA described are consistent with best 
practices in human capital management. 

Conclusions: 

VA has taken steps to improve its management of the VR&E program--
ultimately with the goal of helping more veterans with disabilities 
attain suitable employment--but its ongoing efforts may not 
sufficiently address certain key management issues. With respect to 
performance measures, VA's efforts to address problems with its 
existing rehabilitation rate measure may actually result in the 
adoption of a less transparent and meaningful outcome measure, as it 
is not clear how measuring the number of positive outcomes will 
provide the context necessary to understand program success. 
Furthermore, while VA plans to collect more information about longer-
term outcomes, absent a performance measure VA may miss an opportunity 
to hold staff accountable and make adjustments to the program to 
promote success in maintaining employment--consistent with VR&E's 
central mission. In addition, while VA's ongoing customer satisfaction 
survey has the potential to provide valuable information, without 
performing additional analyses to determine whether responses actually 
reflect the experiences of VR&E participants overall, the agency 
cannot be sure that its survey will ultimately produce sound results 
for informing program management and training investments. 

VA has also made strides related to workload management and training, 
but remaining shortcomings can negatively affect services provided to 
veterans. Because VA's formula for allocating staff among regional 
offices rewards higher-performing offices, which can make it harder 
for lower-performing offices to improve, but does not consider 
educational and vocational counseling cases, which are concentrated in 
certain regions, some offices' ability to serve veterans may be 
negatively affected. Lacking information on the varied approaches that 
regional offices are using to manage workloads also undermines VA's 
ability to help offices optimize their use of staff to serve veterans 
in the most effective manner. Finally, training of VR&E staff is an 
area in which VA has made substantial improvements, but absent 
additional training on job placement and accommodations, counselors 
may be hindered in their ability to help veterans find and maintain 
employment. 

Recommendations for Executive Action: 

The Secretary of Veterans Affairs should direct the Under Secretary 
for Benefits to take the following actions: 

1. In any revised set of national and regional performance measures 
for the VR&E program, include measures of (a) the proportion of 
program participants successfully rehabilitated into employment, and 
(b) the proportion of participants who obtain other benefits from VR&E 
services. 

2. Develop new measures of long-term employment that go beyond the 
minimum 60 days of post-placement monitoring that is currently 
required. In developing measures, consider the feasibility of using 
results from planned post-closure surveys of veterans as a data source. 

3. Consistent with generally accepted survey practices and as 
warranted by survey response rates, conduct nonresponse analysis of 
the results of VA's ongoing Voice of the Veteran customer satisfaction 
surveys. 

4. In revisiting VA's formula for allocating VR&E staff among the 
regional offices, (a) assess the inclusion of factors related to 
regional office performance and if warranted remove them from the 
formula, and (b) assess the exclusion of any factor related to the 
number of educational counseling cases in each regional office and If 
warranted add such a factor. 

5. Collect information on the regional offices' approaches for 
managing their VR&E workloads, assess the advantages and disadvantages 
of these approaches, and use the results of this assessment to provide 
guidance to the offices on potential best practices or options to 
consider. 

6. Provide additional training to all individual VRCs on job placement 
strategies and workplace accommodations, potentially as part of the 
effort to develop a competency-based training approach. 

Agency Comments and Our Evaluation: 

We provided a draft of this report to VA for review and comment. In 
its written comments, produced as appendix IV in this report, VA 
agreed with our conclusions and generally concurred with our 
recommendations. Specifically, VA concurred with 5 of our 6 
recommendations, and noted steps it plans to take to address them, as 
follows. 

* Regarding our recommendation to measure the proportions of VR&E 
participants achieving employment and other benefits, VA said that 
while ratio-based metrics will not be included in its revised employee-
level performance measures, they will be included in the revised 
national-level measures. 

* Regarding our recommendation to conduct nonresponse bias analysis of 
its Voice of the Veteran survey results, VA said that--if funding 
permits--it will modify its survey administration contract to include 
such an analysis. 

* With regard to our recommendation to assess certain aspects of its 
formula for allocating staffing levels across regional offices, VA 
said it plans to revisit the formula and make any changes necessary. 

* With regard to our recommendation to assess regional offices' 
workload management approaches, VA said it plans to collect and 
analyze data on the various approaches and inform the regions about 
best practices. 

* With regard to our recommendation to close certain gaps in staff 
training, VA said it plans to develop and make available to staff new 
training on employment and accommodations. 

VA agreed in principle with our recommendation to develop performance 
measures related to VR&E participants' longer-term employment success. 
Specifically, VA agreed there is value in looking at long-term 
outcomes, and noted its ongoing longitudinal study of a sample of 
participants and its plans to collect data on veterans' post-closure 
service needs. However, VA also pointed out that VR&E counselors 
already work with veterans for many years, including up to 18 months 
following training completion. As such, VA indicated it does not see 
the value in directing limited resources to implementing and executing 
additional post-placement measures or services that will extend 
program participation and may create program dependencies. We 
acknowledge that VA in some cases monitors veterans for 6 months or 
more after they are placed in a job (as we note on page 24 of the 
report, this occurred in over one-third of the cases we examined). We 
also recognize the potential challenges of additional post-employment 
monitoring. At the same time, we believe that unless VA holds itself 
accountable for the longer-term outcomes of all participants who get 
hired into a suitable job, it may miss opportunities to make 
adjustments that better promote job maintenance. We encourage VA to 
explore cost-effective means of checking on veterans' employment 
status at some point more than 60 days after job placement. We did not 
intend and do not believe this measure would necessarily entail 
extending program services or participation beyond current program 
practices. VA also provided technical comments, which we incorporated 
in our report as appropriate. 

We are sending copies of this report to appropriate congressional 
committees, the Secretary of Veterans Affairs, and other interested 
parties. In addition, this report will also be available at no charge 
on GAO's website at [hyperlink, http://www.gao.gov]. 

If you or your staff have any questions about this report, please 
contact me at (202) 512-7215 or bertonid@gao.gov. Contact points for 
our Offices of Congressional Relations and Public Affairs may be found 
on the last page of this report. GAO staff members who made key 
contributions to this report are listed in appendix V. 

Signed by: 

Daniel Bertoni: 
Director, Education, Workforce and Income Security Issues: 

[End of section] 

Appendix I: Objectives, Scope, and Methodology: 

The objectives of this report were to examine (1) the outcomes for 
veterans seeking employment through the Vocational Rehabilitation and 
Employment (VR&E) program, and (2) the progress the Department of 
Veterans Affairs (VA) has made in addressing critical management 
issues. To address these objectives, we reviewed relevant federal 
laws, regulations, and guidance, as well as VA documentation related 
to the VR&E program and recent management initiatives. We also 
reviewed several external studies and evaluations of the VR&E program, 
and determined that their methodologies were sufficient to allow us to 
report certain findings from these studies. We reviewed various 
criteria for effective program management, including in the areas of 
performance management and staff training. We analyzed VA 
administrative data on VR&E participants who applied to the program 
between fiscal years 2003 and 2012 (see below for more information on 
our efforts to analyze and ensure the reliability of these data), and 
interviewed VA central office staff responsible for the VR&E program, 
VR&E managers and staff in VA regional benefits offices (see below for 
details on office selection), and representatives of the Disabled 
American Veterans and the Paralyzed Veterans of America. Finally, to 
obtain input from veterans, we conducted phone interviews with 
veterans who had participated in the VR&E program (see below for 
details on how we developed our survey and selected veterans) and 
analyzed data from a VA customer satisfaction survey, after reviewing 
the methods VA used to develop and pilot the survey (see below for 
more detail on our assessment of the survey). 

Analysis of VA Administrative Data: 

We analyzed data from VA's Corporate WINRS (CWINRS) [Footnote 64] 
system, which VA uses to track veteran cases through the VR&E process 
and manage program costs.[Footnote 65] In our analysis, we included 
veterans who initially applied to the VR&E program between fiscal 
years 2003--the earliest year for which reliable data are available--
and 2012. Unless otherwise noted, we included only veterans who 
received VR&E services designed to help veterans prepare for and 
obtain suitable jobs, and not those who received other services--such 
as educational and vocational counseling--which are also provided by 
VR&E staff. We analyzed data on the characteristics of VR&E 
participants (e.g., age, type of disability); the services they 
received from the VR&E program (e.g., evaluation, training); and their 
outcomes. 

To assess outcomes, we placed VR&E participants who received 
employment-related services into one of four outcome categories: (1) 
rehabilitated to employment, including those who at least once had a 
case status of rehabilitated associated with achieving employment; (2) 
rehabilitated to independent living, including those who never had an 
employment-related rehabilitation but had least one rehabilitated 
status associated with achieving an independent living goal; (3) 
discontinued, including those who never had any rehabilitated status, 
and whose last status as of the end of fiscal year 2012 was 
discontinued;[Footnote 66] and (4) still participating, including 
those who never had a rehabilitated status and whose last status as of 
the end of fiscal year 2012 indicated some type of ongoing 
participation in the program. (See appendix II for a description and 
detailed results of our analysis of the association between VR&E 
participants' outcomes and their characteristics and services 
received.) 

We found the CWINRS data to be sufficiently reliable for the purposes 
of our report. To assess the reliability of the data, we reviewed 
documentation on the system, interviewed VA officials, and performed 
electronic testing. VA officials told us that CWINRS became a single, 
national database during 2002, and that prior data are from separate, 
not necessarily uniform, systems operated by individual regional 
offices and are therefore less reliable. Based on this information, we 
chose to analyze data only for fiscal years 2003 through 2012. Also, 
through electronic testing we identified potential errors with outcome 
data for 1 percent of cases that received some employment-related 
services--for example, cases closed as rehabilitations for reasons 
which could not logically be associated with a rehabilitation. Because 
this made it impossible for us to assign these cases to one of our 
four outcome categories, we excluded them from our analysis of 
veterans' outcomes. Finally, VA officials told us there may be some 
inaccuracies in data related to the status of cases--in particular for 
data prior to fiscal year 2008--because of a programming design flaw 
that allowed correct case status data in CWINRS to be inadvertently 
overridden with obsolete data from another VA system. Therefore, we 
use approximations rather than exact numbers when reporting the number 
of separate occasions on which veterans received certain services 
through the VR&E program, such as evaluation and planning assistance, 
job ready services, and rehabilitation to employment. 

Regional Office Interviews: 

To obtain perspectives on the challenges VR&E staff face in helping to 
rehabilitate veterans, we interviewed VR&E managers and staffs at 8 of 
VA's 57 regional offices.[Footnote 67] We selected offices to achieve 
diversity among offices in the following characteristics: 

* Geography: We selected at least one office in each of the four 
Veterans Benefits Administration areas. 

* Workload: Based on VR&E administrative data, we ranked offices by 
the number of initial fiscal year 2011 VR&E applications filed from 
largest to smallest numbers of applications. 

* Outcomes: Based on VA's ASPIRE VR&E performance dashboard, we ranked 
offices by their Employment Rehabilitation Rate[Footnote 68] in 
September 2012 from highest to lowest percentage rate. 

* Timeliness: Based on VA's ASPIRE VR&E performance dashboard, we 
ranked offices according to the average days required to complete the 
evaluation and planning phase during September 2012 from lowest to 
highest. 

* Unemployment: We considered the November 2012 unemployment rate for 
each office's state, as reported by the Bureau of Labor Statistics. 

See figure 8 for details about the characteristics of each regional 
office we contacted. 

Figure 8: Regional Offices Contacted and Their Characteristics: 

[Refer to PDF for image: illustrated U.S. map] 

Regional office contacted: Hartford; 
Area: Eastern; 
VR&E applications, FY 2011: 39th; 
Employment rehabilitation rate: 6th; 
Evaluation and planning timeliness: 10th; 
State unemployment Nov. 2012[A]: 8.8%. 

Regional office contacted: Manchester; 
Area: Eastern; 
VR&E applications, FY 2011: 51st; 
Employment rehabilitation rate: 51st; 
Evaluation and planning timeliness: 12th; 
State unemployment Nov. 2012[A]: 5.6%. 

Regional office contacted: Washington; 
Area: Southern; 
VR&E applications, FY 2011: 25th; 
Employment rehabilitation rate: 41st; 
Evaluation and planning timeliness: 29th; 
State unemployment Nov. 2012[A]: 8.4%. 

Regional office contacted: Winston-Salem; 
Area: Southern; 
VR&E applications, FY 2011: 6th; 
Employment rehabilitation rate: 31st; 
Evaluation and planning timeliness: 21st; 
State unemployment Nov. 2012[A]: 9.1%. 

Regional office contacted: Houston; 
Area: Central; 
VR&E applications, FY 2011: 5th; 
Employment rehabilitation rate: 8th; 
Evaluation and planning timeliness: 14th; 
State unemployment Nov. 2012[A]: 6.2%. 

Regional office contacted: Milwaukee; 
Area: Central; 
VR&E applications, FY 2011: 24th; 
Employment rehabilitation rate: 7th; 
Evaluation and planning timeliness: 18th; 
State unemployment Nov. 2012[A]: 6.7%. 

Regional office contacted: Oakland; 
Area: Western; 
VR&E applications, FY 2011: 9th; 
Employment rehabilitation rate: 52nd; 
Evaluation and planning timeliness: 27th; 
State unemployment Nov. 2012[A]: 9.8%. 

Regional office contacted: San Diego; 
Area: Western; 
VR&E applications, FY 2011: 3rd; 
Employment rehabilitation rate: 45th; 
Evaluation and planning timeliness: 44th; 
State unemployment Nov. 2012[A]: 9.8%. 

Source: VA and Bureau of Labor Statistics data; National Atlas of the 
United States (map). 

Note: Rankings are out of 58 offices for which VA reported data. VA 
reported data separately for the Cheyenne, Wyoming, office, which is 
considered part of the Denver regional office. 

[A] We used state unemployment rates as a proxy for unemployment in a 
regional office's jurisdiction. Some regional offices have 
jurisdictions covering part of a state; for example, 4 states have 
more than one office each. The national unemployment rate for November 
2012 was 7.7 percent, according to the Bureau of Labor Statistics. 

[End of figure] 

We visited the regional offices in Washington, D.C., and Oakland, 
California, and conducted telephone interviews with VR&E managers and 
staff at the other six offices. At each of the eight offices, we 
interviewed the VR&E Officer (and one or more Assistant VR&E Officers 
at four offices); at least three VRCs; and at least one EC.[Footnote 
69] At our in-person visits to Washington and Oakland, we also 
interviewed representatives of the Disabled American Veterans and 
California Department of Veterans Affairs. Our interview results 
cannot be generalized to all regional offices' VR&E operations. 

Interviews with Veterans: 

We conducted phone interviews with 17 veterans who had participated in 
the VR&E program, including 9 who had discontinued from the program 
and 8 who had achieved rehabilitation to employment. We pretested our 
interview questions twice with each group, and used the results of the 
pretests in our final analysis. We asked questions about the veterans' 
employment, their interactions with their VR&E counselors, and their 
satisfaction with various VR&E services received. We randomly selected 
veterans for interviews, but cannot generalize the results of the 
interviews to the overall population of VR&E participants. To select 
our sample of veterans, we used CWINRS data to identify VR&E 
participants who had received some employment-related services and had 
either discontinued or achieved an employment-related rehabilitation 
between April 1, 2012 and September 30, 2012. We attempted to contact 
49 discontinued veterans and 32 rehabilitated veterans after sending 
an advance informational email. These contact attempts resulted in 
complete interviews with 17 veterans. 

Analysis of Voice of the Veteran Survey: 

We obtained aggregate results from the pilot of VA's Voice of the 
Veteran customer satisfaction survey directly though VA's dashboard--a 
central database that contains the agency's performance metrics and 
customer satisfaction reports. This survey was administered to random 
samples of veterans from different customer populations. Each sample 
received a questionnaire reflecting the VA benefits and services they 
received. We obtained data from the three questionnaires targeted to 
VR&E participants: the Enrollment survey, the Servicing survey, and 
the Escaped Beneficiary survey. All three VR&E pilot surveys were 
fielded from October 2012 to January 2013. Table 2 provides background 
information on each of these surveys, including a description of the 
survey's purpose and target sample population. 

Table 2: Description of Vocational Rehabilitation and Employment's 
Voice of the Veteran Pilot Surveys: 

VR&E pilot survey: Enrollment survey; 
Survey purpose: Obtained veterans' views on benefit eligibility and 
application process, benefit entitlement, benefit information, and VA 
personnel; 
Target population: Veterans who had an initial meeting with their VR&E 
counselor and were granted a decision regarding their entitlement in 
the 60 days before the survey field period. This includes those who 
(1) apply and attend the initial appointment to the program and pursue 
(entitled), (2) apply and attend the initial appointment to the 
program and do not pursue (entitled), and (3) apply and attend the 
initial appointment to the program (not entitled). 

VR&E pilot survey: Servicing survey; 
Survey purpose: Obtained veterans' views on benefit entitlement, 
benefit information, and VA personnel; 
Target population: Veterans who have entered and been enrolled in one 
of the five tracks for at least 60 days before the survey field 
period. This may include veterans who (1) are currently participating, 
(2) been rehabilitated, (3) did not fully complete the program, and 
(4) have reached maximum rehabilitation gain and could not proceed in 
the program. 

VR&E pilot survey: Escaped Beneficiary survey; 
Survey purpose: Obtained veterans' views on benefit entitlement, 
benefit information, VA personnel, and the reasons for deciding not to 
complete the benefit application process and continue to pursue the 
benefit; 
Target population: Veterans who dropped out of the program prior to 
completing a rehabilitation plan. This may include applicants who (1) 
never attended the initial meeting with a counselor, (2) were 
determined to be entitled and did not complete a rehabilitation plan, 
and (3) started but did not complete rehabilitation. 

Source: VA survey documentation. 

[End of table] 

the Voice of the Veteran surveys, and our review of documentation of 
the surveys’ methodology, we concluded that these three VR&E surveys 
were generally developed and piloted in accordance with generally 
recognized survey research practices, and that the results we cite in 
this report are sufficiently reliable for our analysis. We used 
standards and guidelines issued by the Office of Management and Budget 
(OMB) to assess the methods VA used to design and administer the 
surveys.[Footnote 7] In addition, we reviewed survey guidelines of 
generally recognized professional organizations such as the American 
Association for Public Opinion Research and the American Statistical 
Association.[Footnote 8] 

However, we concluded that the pilot survey results cited should be 
interpreted as representative of only those veterans responding to the 
survey. Response rates for the VR&E surveys were low—between 16 
percent and 25 percent—and fell below the expected 30 percent rates 
specified in VA’s sample design. Table 3 provides more information on 
sample outcomes. While response rates may not be highly correlated 
with the presence of nonresponse bias and are not considered, by 
themselves, to be sufficient indicators of the quality of survey 
results, response rates are a useful indicator of the risk of 
nonresponse bias. According to OMB’s guidelines, a survey’s response 
rate is a valuable data quality and performance indicator, and is 
associated with the generalizability of a survey’s results. In 
addition, OMB’s guidelines state that a nonresponse analysis should be 
performed when a survey’s unit response rate is below 80 percent, to 
identify the possibility of bias in a survey’s results, and that 
agencies should appropriately measure, adjust for, report, and analyze 
nonresponse to assess its effect on data quality. Because VA did not 
perform a nonresponse analysis to determine if systematic differences 
exist between respondents and nonrespondents, it is not possible to 
generalize the results to the broader population of VR&E participants. 
Also, because the effects of any nonresponse bias on data quality were 
not assessed, it is not known if any systematic differences exist 
between nonrespondents and respondents. 

Table 3: Sample Outcomes for Vocational Rehabilitation and 
Employment's Voice of the Veteran Pilot Surveys: 

Enrollment survey; 
Veteran population (annual): 77,998; 
Veterans sampled: 4,599; 
Target number of survey completions[A]: 3,000; 
Usable surveys completed: 939; 
Proportion of target completed[B]: 31%; 
Response rate[C]: 20%. 

Servicing survey; 
Veteran population (annual): 121,722; 
Veterans sampled: 8,781; 
Target number of survey completions[A]: 3,000; 
Usable surveys completed: 2,161; 
Proportion of target completed[B]: 72%; 
Response rate[C]: 25%. 

Escaped beneficiary survey; 
Veteran population (annual): 11,700; 
Veterans sampled: 5,000; 
Target number of survey completions[A]: 1,500; 
Usable surveys completed: 817; 
Proportion of target completed[B]: 54%; 
Response rate[C]: 16%. 

Source: VA Voice of the Veteran Survey documentation. 

[A] The "target number of survey completions" was identified in VA's 
initial sample design, and was based on VA's expected response rates. 

[B] The "proportion of target completed" was calculated by dividing 
the number of "usable surveys completed" by the "target number of 
survey completions." 

[C] The "response rate" was calculated by dividing the number of 
"usable surveys completed" by the number of "veterans sampled." 

[End of table] 

[End of section] 

Appendix II: Statistical Analysis of Administrative Data on Vocational 
Rehabilitation and Employment Outcomes: 

We used a statistical method known as survival analysis to estimate 
Vocational Rehabilitation and Employment (VR&E) participants’ 
likelihood of achieving success, their program completion times, and 
the factors that were associated with rehabilitation. Survival 
modeling methods provide a simple and efficient way to estimate the 
rate of event occurrence when the duration times of the events are “
censored,” meaning that they are unobserved over some portion of the 
time period of interest. For example, a rehabilitation time would be 
censored if a veteran had been in a program for 2 years but had 
neither dropped out nor finished the program at the time of analysis. 
Although the veteran’s rehabilitation time is at least 2 years, the 
ultimate time to rehabilitation or discontinuation is not yet known 
(censored). We used a specific type of survival method, Cox regression 
models, to identify predictors of program completion-—or what the 
Department of Veterans Affairs (VA) defines as “rehabilitation”-—and 
estimate the rate at which rehabilitation occurred at various times 
after application among various groups of veterans. 

Cox models allowed us to analyze the data without needing to make 
strong assumptions about the rehabilitation process. We did not need 
to assume that the likelihood or rate of rehabilitation changed over 
time in specific ways, as we would with other survival methods. The 
flexibility of Cox models comes at some cost of interpretation, 
however. The mathematics of Cox models make it more difficult to 
estimate the probability of rehabilitation by certain follow-up times 
(known as “failure probabilities”). To avoid the complex computation 
required for these estimates, we generally report simpler “Kaplan-Meier”
estimates of rehabilitation probabilities in the body of the report-—a 
form of survival analyses that allows us to report on likelihood of 
program completion, but without controlling for other covariates. To 
corroborate these results, we use Cox models to estimate the “hazard 
rate” of rehabilitation in this appendix, which is loosely equal to 
the rate of rehabilitation at a certain time after starting a VR&E 
program, while controlling for other factors. The results of this 
analysis are presented in this appendix and essentially support the 
same substantive findings about factors associated with the chance of 
rehabilitation discussed in the body of this report. 

Source Data and Measurement of Key Variables: 

For the survival analysis using Cox models and Kaplan-Meier methods, 
we used data from the VR&E program’s Corporate WINRS system, which 
includes archived data on program participants’ demographic 
characteristics, services received, and outcomes achieved. We selected 
data on veterans who began their first VR&E programs in fiscal year 
2003 through fiscal year 2012. We limited our analysis to this time 
period after VA identified reliability problems with older data. 
Appendix I describes the source data and their reliability in more 
detail. 

In initial exploratory analysis, we considered various factors that 
could be associated with successful program completion and duration. 
These factors (which we also may refer to as covariates, variables, or 
predictors) included age, sex, education, branch of service, number of 
counselors, number of discontinuations, severity of disability, type 
of disability, year of first application to the program, VR&E services 
provided (“service track”), and the regional offices that managed 
veterans’ cases. 

A key strength of the VR&E administrative data is its volume. The 
total number of veterans available for analysis, using the population 
defined above, was 133,679. Table 4, below, shows how the data were 
distributed across years of program entry and program outcomes. The 
large sample size available for analysis allowed us to estimate valid 
and precise relationships between completion times and a large number 
of covariates. We confirmed the adequacy of the sample by examining 
the cell sample sizes in tabulations of potential covariates. 

Table 4: Sample Size by Fiscal Year of Program Entry and Outcomes: 

Fiscal Year of First Application: 2003; 
Number: 16,726; 
Percent of Sample: 12.5%. 

Fiscal Year of First Application: 2004; 
Number: 14,587; 
Percent of Sample: 10.9%. 

Fiscal Year of First Application: 2005; 
Number: 14,653; 
Percent of Sample: 11.0%. 

Fiscal Year of First Application: 2006; 
Number: 13,636; 
Percent of Sample: 10.2%. 

Fiscal Year of First Application: 2007; 
Number: 14,667; 
Percent of Sample: 11.0%. 

Fiscal Year of First Application: 2008; 
Number: 16,078; 
Percent of Sample: 12.0%. 

Fiscal Year of First Application: 2009; 
Number: 14,490; 
Percent of Sample: 10.8%. 

Fiscal Year of First Application: 2010; 
Number: 11,264; 
Percent of Sample: 8.4%. 

Fiscal Year of First Application: 2011; 
Number: 10,220; 
Percent of Sample: 7.7%. 

Fiscal Year of First Application: 2012; 
Number: 7,358; 
Percent of Sample: 5.5%. 

Program Outcome: Rehabilitated to employment; 
Number: 38,664; 
Percent of Sample: 28.9%. 

Censored: Program in progress; 
Number: 68,462; 
Percent of Sample: 51.2%. 

Censored: Discontinued; 
Number: 23,271; 
Percent of Sample: 17.4%. 

Censored: Transferred to Independent Living track and achieved 
rehabilitation; 
Number: 1,351; 
Percent of Sample: 1.0%. 

Outcome not recorded; 
Number: 1,931; 
Percent of Sample: 1.4%. 

Total: 
Number: 133,679; 
Percent of Sample: 100.0%. 

Source: GAO analysis of VR&E administrative data. 

Note: All percentages do not add to 100 due to rounding. 

[End of table] 

We assessed the degree of correlation among the initial list of 
covariates, in order to identify those that would be redundant or 
introduce the possibility of spurious estimates. Most of the potential 
covariates were not strongly intercorrelated, except for the number of 
counselors and discontinuations, the number of counselors and regional 
offices, and age and era of service. The pair-wise correlations for 
these covariates ranged from 0.37 to 0.52. Including all of these 
covariates in our model would have inflated the estimated 
relationships between them and program completion times. Therefore, we 
included age and number of counselors in the final list of covariates, 
and excluded era of service, number of discontinuations, and number of 
offices from our model. 

Measuring most of the covariates of interest was straightforward, and 
can be determined for a subset of meaningful predictors, as shown in 
Table 6 below. However, measuring certain covariates—-such as regional 
offices, certain VR&E services, and disability type—-required more 
complex measurement decisions. 

Because veterans can be served by a number of regional offices over 
time, measuring the offices serving each veteran is challenging. In 
our data, veterans’ cases were managed by more than 4,100 permutations 
of regional offices. The large number of permutations made it 
difficult to meaningfully analyze variation across offices, and, in 
any case, produced sample sizes that were too small for modeling 
purposes. To simplify the measurement, we focused on the last office 
where each veteran was enrolled, as a measure of regional office at 
program completion. Because veterans could have discontinued their 
programs multiple times and could have been served by as many as six 
offices, the last office was a logical proxy for the office that could 
have influenced the final outcome most strongly. 

Because we selected only veterans from the VA administrative data who 
received employment-related services, this group received job training 
or job placement assistance. We created a single measure of services 
received by creating categories for specific services related to 
employment, such as job placement services, and specific types of job 
training received, such as high school or college education. Table 5 
below identifies the services we measured. 

Finally, veterans could have been diagnosed with one or more 
disabilities, such as mental health conditions, ear disorders, and/or 
cardiovascular disorders. This allows for a potentially large number 
of unique combinations of disorders, which makes it difficult to 
create mutually exclusive and exhaustive groups. To avoid this 
problem, we compared veterans who had each type of disorder to 
veterans who did not have the disorder, using binary indicators for 
the presence or absence of each diagnosis. While the possibility of 
multiple disabilities makes these indicators mutually inclusive, our 
measurement scheme provided a concise way to collapse the many unique 
combinations of disorders. 

The purpose of survival modeling is to estimate the probability of an 
event by various follow-up times or the average rate of event 
occurrence at any one follow-up time. Because these two quantities are 
mathematically equivalent, we used two different methods that produce 
similar results but vary in their ease of interpretation.
In the body of this report, we provide Kaplan-Meier estimates of the 
probability of rehabilitation by 8 years after the time of first 
application, known as a “failure” probability in survival analysis. 
Let tj denote one of j = 1, 2, ..., k rehabilitation times observed in 
the data. Then the Kaplan-Meier failure estimate at tj is given by: 

[Refer to PDF for formula] 

where nj is the number of veterans who are “at risk” of rehabilitation 
and dj is the number of rehabilitations observed at tj. We estimated 
failure probabilities for various subgroups of veterans, in order to 
describe the association between these factors and the chance of 
rehabilitation by 5 and 8 years. 

To corroborate these associations and control for alternative 
variables that could account for them, we used multivariate survival 
models-—and the Cox Model for Proportional Hazards, in particular-—to 
estimate the average rate of event occurrence at any one time in an 
observation period, also known as the “hazard rate.”[Footnote 9] 
1
In our analysis, the event of interest is rehabilitation, and the time 
period of observation is fiscal year 2003 through fiscal year 2012. In 
our analysis, we defined censored outcomes as a veteran’s remaining in 
the program at the end of fiscal year 2012 or having discontinued from 
a program without yet re-applying (see table 5 above). 

The formula for our Cox regression model was: 

[Refer to PDF for formula] 

where ?i(t, xi) is the hazard rate for veteran i at time t, xi = xi1 
,..., xip is a vector of p covariates, and xib is a linear combination 
of covariates and coefficients, with no intercept term. As described 
above and in table 6, the covariates included: age, sex, job training 
services, the interaction of sex and job training services, education, 
branch of service, number of counselors, severity of disability, and 
disability type. The term ?0(t) is the “baseline hazard rate,” or the 
rate implied when all covariates equal zero. (Note that the equation 
simplifies to ?0(t) when this is true.) 

The Cox Model assumes that the hazard rates are proportional across 
values of the covariates throughout the observation period. If the 
hazard in one group exceeds that of a reference group, the model 
assumes that the proportional difference in rates will remain 
constant, and that the difference in hazard rates across groups does 
not decline, increase, or reverse direction over time. For example, 
the model assumes that the rehabilitation rate for veterans with 
mental health conditions differs from the rate for veterans without 
this type of condition by a constant amount throughout the observation 
period. 

However, because a particular data generation process may not satisfy 
the proportional hazards assumption, we plotted the estimated hazards 
for various values of each covariate over time and formally tested the 
assumption for each covariate using statistical tests based on the 
model’s Shoenfeld residuals. We found a non-zero slope in a 
generalized linear regression of the Schoenfeld residuals on functions 
of time for several variables. A non-zero slope indicates that the 
hazards are not proportional across values of the covariates over 
time.[Footnote 10] A global test across all variables further 
suggested that the hazard function was not proportional. 

To identify the most serious departures from the proportional hazard 
assumption, we plotted the cumulative hazard function over time for 
the covariates that failed the Schoenfeld test. These plots suggested 
that the last office and first year of application covariates had 
strongly disproportional hazards. The remaining variables only failed 
to meet the proportional hazard assumption in a category coded for “
missing data,” and the other variables that appeared to be 
disproportional were not meaningfully associated with rehabilitation 
times. 

To adjust for most seriously non-proportional hazards, we stratified 
the model by last office and first year of application. Stratification 
allowed us to control for these two factors while also allowing their 
hazard functions to vary over time across offices and years of 
application. After stratifying by last office and first year of 
application, the proportional hazard assumption for the model was more 
consistent with our data. 

Results: 

Our analysis identified a number of factors that had strong and 
statistically significant associations with rehabilitation times, as 
shown in Table 5. 

Veterans took more time to complete their programs if multiple 
counselors worked with them. Average completion times increased at a 
higher rate as the number of counselors increased from 1 to 5 than 
from 6 to 9 or more. For example, veterans with 2 counselors had 
average rehabilitation times that were 29 percent slower, on average, 
than veterans with 1 counselor. In contrast, veterans with 9 or more 
counselors had average rehabilitation times that were about 1 percent 
slower than veterans with 8 counselors. In exploratory analysis, we 
found that veterans who more often discontinued and re-entered the 
program and who were served by multiple regional offices were more 
likely to work with multiple counselors, and that discontinuing more 
frequently was associated with slower rates of rehabilitation. 
However, when we estimated Cox models that controlled for the number 
of times discontinued and the number of regional offices used, we 
found that the number of counselors still had a meaningful association 
with success. Nevertheless, a reciprocal relationship, in which longer 
program times are associated with more counselors, may also explain 
this pattern. 

The presence of mental health conditions substantially decreased 
completion rates. Veterans with mental health conditions completed 
their programs at a rate that was about 27 percent slower on average 
than veterans who did not have mental health conditions.[Footnote 11] 

As expected, the type of services veterans received was among the 
strongest predictors of completion times. Veterans who received job 
placement services finished their programs about 4.5 times more 
quickly than veterans who were pursuing an undergraduate education. 
This is consistent with the nature of VR&E’s job placement services, 
which focus on veterans who already have the skills to pursue 
employment. Veterans who received apprenticeships, non-vocational 
training, unpaid work at a federal agency, vocational or technical 
education, or non-degree college credit hours finished their programs 
about 1.6 to 2.4 times more quickly than veterans pursuing an 
undergraduate college degree. In contrast, the average completion rate 
for veterans pursuing a graduate degree was only 28 percent quicker 
than for veterans pursuing an undergraduate degree. As a whole, these 
results suggest that veterans pursuing undergraduate or graduate 
degrees require more time to complete their programs than veterans 
receiving other training. 

The amount of prior education was associated with faster program 
completion rates. As compared to veterans with only a high school 
degree, veterans with some college, an undergraduate degree, or a 
graduate degree had completion rates that were 1.2 to 1.5 times 
faster, on average. Because these estimates control for the type of 
services received and the type and severity of disabilities, veterans 
with more prior education appear to have finished more quickly 
regardless of the training they received or the nature of their 
disabilities. 

Finally, veterans with disability ratings of less than 100 percent 
finished their programs more quickly than veterans with disability 
ratings of 100 percent, with completion rates that were about 1.5 
times more quickly on average.[Footnote 12] This pattern may reflect 
the fact that veterans with more severe disabilities face greater 
challenges to timely program completion, even holding constant factors 
such as type of training, prior education, and age. 

Table 5: Estimated Hazard Ratios of Time to Vocational Rehabilitation 
and Employment Program Completion: 

Variable: Prior Education; High School; 
Hazard Ratio: 1.000 (Referent); 
95% Lower Bound: [Empty]; 
95% Upper Bound: [Empty]. 

Variable: Prior Education; Less than High School; 
Hazard Ratio: 0.923[C]; 
95% Lower Bound: 0.812; 
95% Upper Bound: 1.05. 

Variable: Prior Education; Some College; 
Hazard Ratio: 1.230[C]; 
95% Lower Bound: 1.200; 
95% Upper Bound: 1.261. 

Variable: Prior Education; College; 
Hazard Ratio: 1.449[D]; 
95% Lower Bound: 1.394; 
95% Upper Bound: 1.506. 

Variable: Prior Education; Graduate; 
Hazard Ratio: 1.461[C]; 
95% Lower Bound: 1.330; 
95% Upper Bound: 1.606. 

Variable: Prior Education; Missing Data; 
Hazard Ratio: 2.585[C]; 
95% Lower Bound: 2.503; 
95% Upper Bound: 2.669. 

Variable: Number of Case Managers; 1; 
Hazard Ratio: 1.000 (Referent); 
95% Lower Bound: [Empty]; 
95% Upper Bound: [Empty]. 

Variable: Number of Case Managers; 2; 
Hazard Ratio: 0.712[C]; 
95% Lower Bound: 0.677; 
95% Upper Bound: 0.748. 

Variable: Number of Case Managers; 3; 
Hazard Ratio: 0.471[C]; 
95% Lower Bound: 0.447; 
95% Upper Bound: 0.495. 

Variable: Number of Case Managers; 4; 
Hazard Ratio: 0.336[C]; 
95% Lower Bound: 0.318; 
95% Upper Bound: 0.354. 

Variable: Number of Case Managers; 5; 
Hazard Ratio: 0.230[C]; 
95% Lower Bound: 0.216; 
95% Upper Bound: 0.244. 

Variable: Number of Case Managers; 6; 
Hazard Ratio: 0.159[C]; 
95% Lower Bound: 0.147; 
95% Upper Bound: 0.172. 

Variable: Number of Case Managers; 7; 
Hazard Ratio: 0.115[C]; 
95% Lower Bound: 0.102; 
95% Upper Bound: 0.130. 

Variable: Number of Case Managers; 8; 
Hazard Ratio: 0.088[C]; 
95% Lower Bound: 0.071; 
95% Upper Bound: 0.109. 

Variable: Number of Case Managers; 9+; 
Hazard Ratio: 0.073[C]; 
95% Lower Bound: 0.054; 
95% Upper Bound: 0.010. 

Variable: Type of Services Received[A]; Undergraduate; 
Hazard Ratio: 1.000 (Referent); 
95% Lower Bound: [Empty]; 
95% Upper Bound: [Empty]. 

Variable: Type of Services Received[A]; Job Placement Only; 
Hazard Ratio: 4.463[C]; 
95% Lower Bound: 4.263; 
95% Upper Bound: 4.673. 

Variable: Type of Services Received[A]; Missing; 
Hazard Ratio: 1.079[E]; 
95% Lower Bound: 1.012; 
95% Upper Bound: 1.150. 

Variable: Type of Services Received[A]; Graduate School; 
Hazard Ratio: 1.282[C]; 
95% Lower Bound: 1.218; 
95% Upper Bound: 1.349. 

Variable: Type of Services Received[A]; College, Non-Degree, Credit 
Hour; 
Hazard Ratio: 1.679[C]; 
95% Lower Bound: 1.576; 
95% Upper Bound: 1.789. 

Variable: Type of Services Received[A]; Non-standard degree and/or 
voc/tech; 
Hazard Ratio: 2.376[C]; 
95% Lower Bound: 2.291; 
95% Upper Bound: 2.463. 

Variable: Type of Services Received[A]; Nonpaid work fed agency; 
Hazard Ratio: 1.738[C]; 
95% Lower Bound: 1.564; 
95% Upper Bound: 1.930. 

Variable: Type of Services Received[A]; HS/Farm, Special Rehab; 
Hazard Ratio: 1.523[D]; 
95% Lower Bound: 1.187; 
95% Upper Bound: 1.955. 

Variable: Type of Services Received[A]; OJT; 
Hazard Ratio: 3.028[C]; 
95% Lower Bound: 2.671; 
95% Upper Bound: 3.433. 

Variable: Type of Services Received[A]; Apprenticeship; 
Hazard Ratio: 1.635[C]; 
95% Lower Bound: 1.309; 
95% Upper Bound: 2.043. 

Variable: Type of Services Received[A]; Non-Vocational; 
Hazard Ratio: 1.800[C]; 
95% Lower Bound: 1.642; 
95% Upper Bound: 1.973. 

Variable: Type of Services Received[A]; Unpaid OJT, fe, home, 
facility, shelter etc.; 
Hazard Ratio: 1.708[C]; 
95% Lower Bound: 1.442; 
95% Upper Bound: 2.024. 

Variable: Type of Disability[B]; Auditory; 
Hazard Ratio: 1.104[C]; 
95% Lower Bound: 1.081; 
95% Upper Bound: 1.129. 

Variable: Type of Disability[B]; Cardiovascular; 
Hazard Ratio: 1.048[C]; 
95% Lower Bound: 1.023; 
95% Upper Bound: 1.073. 

Variable: Type of Disability[B]; Gynecological; 
Hazard Ratio: 1.089[C]; 
95% Lower Bound: 1.043; 
95% Upper Bound: 1.137. 

Variable: Type of Disability[B]; Infectious/Immune Disease; 
Hazard Ratio: 0.941[E]; 
95% Lower Bound: 0.893; 
95% Upper Bound: 0.990. 

Variable: Type of Disability[B]; Mental health/PTSD; 
Hazard Ratio: 0.731[C]; 
95% Lower Bound: 0.716; 
95% Upper Bound: 0.747. 

Variable: Type of Disability[B]; Neurological; 
Hazard Ratio: 0.948[C]; 
95% Lower Bound: 0.928; 
95% Upper Bound: 0.968. 

Variable: Type of Disability[B]; Respiratory; 
Hazard Ratio: 1.075[D]; 
95% Lower Bound: 1.052; 
95% Upper Bound: 1.099. 

Variable: Type of Disability[B]; Skin; 
Hazard Ratio: 1.073[D]; 
95% Lower Bound: 1.050; 
95% Upper Bound: 1.096. 

Variable: Severity of Disability; Full Disability (100 percent); 
Hazard Ratio: 1.000 (Referent); 
95% Lower Bound: [Empty]; 
95% Upper Bound: [Empty]. 

Variable: Severity of Disability; Partial Disability (0-90 percent); 
Hazard Ratio: 1.492[C]; 
95% Lower Bound: 1.419; 
95% Upper Bound: 1.570. 

Variable: Severity of Disability; PT/Missing; 
Hazard Ratio: 0.760[C]; 
95% Lower Bound: 0.682; 
95% Upper Bound: 0.848. 

Source: GAO analysis of VR&E administrative data. 

Note: Entries are exponentiated coefficients and 95 percent confidence 
intervals from a Cox proportional hazard model of VR&E program 
completion, treating rehabilitation to employment as the outcome of 
interest and all other events as censored. The model included 
additional covariates not listed in this table, as described in the 
body of this appendix. The hazard function was stratified by regional 
office and year of first application. N = 132,042: 

[A] Estimates are summarized for male veterans, due to an interaction 
term in the model for sex and type of services received. The results 
reported here for men are substantially similar to the results for 
both sexes when the interaction term is excluded. 

[B] Estimates apply to the presence or absence of each disorder. 
Because veterans can have one or more disorders, and many combinations 
of disorders are possible, we did not use mutually exclusive and 
exhaustive categories. 

[C] p<0.001: 

[D] p<0.01: 

[E] p<0.05: 

[End of table] 

[End of section] 

Appendix III: Vocational Rehabilitation Counselor Training Gaps 
Identified by VA in 2010 and the Agency's Efforts to Address: 

New VRCs: 

Training gaps: 
How to provide services to veterans recently discharged from the 
military; 
How to provide services to veterans with severe disabilities; 
How to determine when extended evaluation is necessary and conduct 
subsequent individualized extended evaluation plan development; 
How to closeout cases for different veteran outcomes (e.g., for 
veterans who achieved suitable employment or maximum rehabilitation 
gain); 
Courses: 40-hour training course; 
Fiscal year course provided: 2013. 

For new and experienced VRCs: 

Training gaps: How to provide services to veterans with traumatic brain 
injuries (TBI); 
Courses: 1-hour course on mild TBI; 
Fiscal year course provided: 2012.
Courses: 2-hour course on moderate to severe TBI; 
Fiscal year course provided: 2013.
Courses: 2-hour course on vocational implications of TBI; 
Fiscal year course provided: 2013. 

Training gaps: How to provide services to veterans with post-traumatic 
stress disorder (PTSD); 
Courses: 1.5 hour course on PTSD: The VR&E Counselor's Role; 
Fiscal year course provided: 2012.
Courses: Training gaps: 2-hour course on mental health awareness[A]; 
Fiscal year course provided: 2014.
Courses: 3-hour course on mental health techniques[A]; 
Fiscal year course provided: 2014. 

Training gaps: How to develop an independent living (IL) plan; 
Courses: 1-hour course on IL (plan) construction; 
Fiscal year course provided: 2012.
Courses: 1-hour course on IL program guidance[B]; 
Fiscal year course provided: 2013. 

Training gaps: How to conduct a preliminary and/or comprehensive IL 
assessment; 
Courses: 15-hour course on IL; 
Fiscal year course provided: 2014[C]. 

Training gaps: How to input benefit awards into the computer system 
(e.g., inputting the decision to provide a veteran a subsistence 
allowance); 
Courses: 2-hour course on VR&E fiscal accuracy and integrity[D]; 
Two 1-hour courses on VR&E processing of subsistence 
allowance awards[D]; 1-hour course on employment adjustment 
allowances[D]; 
Fiscal year course provided: 2012. 

Source: GAO review of VA-sponsored June 2010 skill gap analysis and 
additional information provided by VA. 

Source: GAO review of VA-sponsored June 2010 skill gap analysis and 
additional information provided by VA. 

[A] The new courses on mental health will address PTSD and other 
mental illnesses and are intended to help increase successful VR&E 
outcomes for veterans with these conditions. 

[B] VA added this course to the fiscal year 2013 curriculum to address 
changes to its guidance for independent living and delivered the 
training in September, 2013. 

[C] VA has developed the course and will assign it as part of the 
fiscal year 2014 curriculum to be completed by the end of the first 
fiscal quarter (i.e., by December 31, 2013). 

[D] VA used existing courses to train VR&E staff on inputting benefit 
awards into its computer system. 

[End of table] 

[End of section] 

Appendix IV: Comments from the Department of Veterans Affairs: 

Department of Veterans Affairs: 
Washington, DC 20420 

December 3, 2013: 

Mr. Daniel Bertoni: 
Director, Education Workforce and Income Security Issues: 
U.S. Government Accountability Office: 
441 G Street, NW: 
Washington, DC 20548: 

Dear Mr. Bertoni: 

The Department of Veterans Affairs (VA) has reviewed the Government 
Accountability Office's (GAO) draft report, "VA Vocational 
Rehabilitation And Employment: Further Performance and Workload 
Management Improvements Are Needed" (GAO-14-61). VA generally agrees 
with GAO's conclusions and concurs with GAO's recommendations and 
concurs in principle with recommendation 2, to the Department. 

The enclosure specifically addresses GAO's recommendations and 
provides technical comments to the draft report. VA appreciates the 
opportunity to comment on your draft report. 

Sincerely, 

Signed by: 

Jose D. Riojas: 
Chief of Staff: 

Enclosure: 

Department of Veterans Affairs (VA) Response to Government 
Accountability Office (GAO) Draft Report: 

"VA Vocational Rehabilitation And Employment: Further Performance and
Workload Management Improvements Are Needed" (GAO-14-61): 

Recommendation 1: In any revised set of VR&E performance measures, 
include measures of (a) the proportion of program participants 
successfully rehabilitated into employment and (b) the proportion of 
participants who obtain other benefits from VR&E services. 

VA Response: Concur. VA agrees that a comprehensive measure of the 
Vocational Rehabilitation and Employment (VR&E) program success would 
include an evaluation of successful outcomes for both employment and 
other benefits or tracks. VA does not plan to continue to measure 
ratio-based performance elements for employee performance, since this 
penalizes employees for closing cases when Veterans make personal 
choices to discontinue participating in their programs. Broader spectrum
performance data which provides the context necessary to evaluate 
program success is more appropriately gathered nationally, as opposed 
to at the employee level. These national key measures are being 
finalized, and VA expects to have them approved for implementation in 
fiscal year (FY) 2015. Target completion date: October 31, 2014. 

Recommendation 2: Develop new measures of long-term employment that go
beyond the minimum 60 days of post-placement monitoring that is 
currently required. In developing measures, consider the feasibility 
of using results from planned post-closure surveys of veterans as a 
data source. 

VA Response: Concur in principle. VA agrees with the importance of 
collecting data on Veterans after case closure. VA is currently 
collecting information on Veterans' long-term outcomes through VR&E's 
Longitudinal Study. The Longitudinal Study will, over a 20-year 
period, track a statistically valid sample of three cohorts that 
participated in the VR&E program during FYs 2010, 2012, and 2014. The 
2013 Annual Report was the first in the Longitudinal Study to present 
self-reported survey data on the employment and standard of living 
outcomes of Veterans who establish a plan of services. VR&E is also 
designing a post-outcome tool that will enable VR&E counselors to 
solicit information from Veterans to determine if there is a need for 
any further services after their cases are closed into either 
rehabilitated or discontinued status. This data and information will 
be valuable for strategic planning and program management. 

VR&E offers extensive individualized services to assist Veterans in 
obtaining and maintaining suitable employment. Unlike many other 
Federal programs, VR&E provides one-on-one vocational counseling, 
training, and job-readiness services typically over multiple years, 
with additional employment services of up to 18 months to ensure that
Veterans are well prepared to get and keep suitable employment 
positions. 

In fact, GAO commented on the lengthy time period that program 
participants spend in the VR&E program. VA therefore, does not see the 
value in directing limited resources to implementing and executing 
additional post-placement measures. Requiring lengthy post-placement 
follow-up services will only further extend the program participation
period and may create problematic program dependencies among some 
participants. 

Recommendation 3: Consistent with generally accepted survey practices, 
and as warranted by survey response rates, conduct nonresponse bias 
analysis of the results of VA's ongoing Voice of the Veteran customer 
satisfaction surveys. 

VA Response: Concur. While VA's Voice of the Veteran Continuous 
Measurement Satisfaction Research program contract does not include a 
task for nonresponse bias analysis, the findings are statistically 
valid. If the program is fully funded in 2014, its option year 
contract will be initiated and VA will include a modification 
requiring the contractor to conduct non response bias analysis of the 
results when needed or required. If funded, the modification is 
expected to be completed by the end of September 2014. Target 
completion date: September 30, 2014. 

Recommendation 4: In revisiting VA's formula for allocating VR&E staff 
among the regional offices, (a) assess the inclusion of factors 
related to regional office performance and if warranted remove them 
from the formula, and (b) assess the exclusion of any factor related 
to the number of educational counseling cases in each regional office 
and if warranted add such a factor. 

VA Response: Concur. At the beginning of each FY, the Veterans Benefits
Administration's (VBA) Office of Field Operations works closely with 
the four area directors to address concerns regarding caseloads. With 
the evolution and completion of the new VR&E staffing model, VBA will 
revisit the metrics used in the resource allocation model (RAM) to 
ensure continued validity and data integrity. Full implementation of 
an updated RAM will occur with the distribution of the FY 2015 budget 
allocation for VBA Target completion date: October 31, 2014. 

Recommendation 5: Collect information on the regional offices' 
approaches for managing their VR&E workloads, assess the advantages 
and disadvantages of these approaches, and use the results of this 
assessment to provide guidance to the offices on potential best 
practices or options to consider. 

VA Response: Concur. VA will work on developing an appropriate 
methodology to collect data from all field offices, engage in data 
collection efforts, analyze the data, and inform field staff of the 
findings with regard to best practices. 

VA anticipates this activity will be completed in the first quarter of 
FY 2015. Target completion date: December 31, 2014. 

Recommendation 6: Provide training to all VRC's and EC's on job 
placement strategies and workplace accommodations, potentially as part 
of the effort to develop a competency-based training approach. 

VA Response: Concur. VR&E has provided several training activities 
focused on job placement and job accommodations, including VR&E 
Training Performance Support System (TPSS), the 2012 Employment 
Coordinators Classroom Training, and the Vocational Rehabilitation 
Counselor (VRC) Gateway Electronic Performance Support System (EPSS), 
and will continue provide training in this area. Training on special
employer incentives (SEI) will be available to all staff in the second 
quarter of FY 2014. Additional training focused on employment and 
accommodations will be developed and available to all staff by the 
last quarter of FY 2014. Target completion date: September 30, 2014. 

[End of section] 

Appendix V: GAO Contact and Staff Acknowledgments: 

GAO Contact: 

Daniel Bertoni, (202) 512-7215 or bertonid@gao.gov: 

Staff Acknowledgments: 

In addition to the contact listed above, individuals making key 
contributions to this report were Michele Grgich (Assistant Director), 
Jessica Botsford, Frederick Caison, Elizabeth Curda, Kevin Daly, Holly 
Dye, Julie DeVault, Brenda Farrell, Alex Galuten, Shannon Grabich, 
Mitch Karpman, Kirsten B. Lauber, Lorin Obler, Carl Ramirez, Almeta 
Spencer, Jeff Tessin, Greg Whitney, and Ashanta Williams. 

[End of section] 

Related GAO Products: 

VA Vocational Rehabilitation and Employment Program: Improved 
Oversight of Independent Living Services and Supports Is Needed. 
[hyperlink, http://www.gao.gov/products/GAO-13-474]. Washington, D.C.: 
June 7, 2013. 

Veterans' Employment and Training: Better Targeting, Coordinating, and 
Reporting Needed to Enhance Program Effectiveness. [hyperlink, 
http://www.gao.gov/products/GAO-13-29]. Washington, D.C.: December 13, 
2012. 

Federal Information System Controls Audit Manual (FISCAM). [hyperlink, 
http://www.gao.gov/products/GAO-09-232G]. Washington, D.C.: February 
2009. 

VA Vocational Rehabilitation and Employment: Better Incentives, 
Workforce Planning, and Performance Reporting Could Improve Program. 
[hyperlink, http://www.gao.gov/products/GAO-09-34]. Washington, D.C.: 
January 26, 2009. 

Multiple Agencies Provide Assistance to Service-disabled Veterans or 
Entrepreneurs, but Specific Needs Are Difficult to Identify and 
Coordination Is Weak. [hyperlink, 
http://www.gao.gov/products/GAO-09-11R]. Washington, D.C.: October 15, 
2008. 

Disabled Veterans' Employment: Additional Planning, Monitoring, and 
Data Collection Efforts Would Improve Assistance. [hyperlink, 
http://www.gao.gov/products/GAO-07-1020]. Washington, D.C.: September 
12, 2007. 

Highlights of a GAO Forum: Modernizing Federal Disability Policy. 
[hyperlink, http://www.gao.gov/products/GAO-07-934SP]. Washington, 
D.C.: August 3, 2007. 

VA Vocational Rehabilitation and Employment: Service Contract 
Management Is Improving, but Challenges Remain. [hyperlink, 
http://www.gao.gov/products/GAO-07-568R]. Washington, D.C.: April 23, 
2007. 

Federal Disability Assistance: Wide Array of Programs Needs to Be 
Examined in Light of 21st Century Challenges. [hyperlink, 
http://www.gao.gov/products/GAO-05-626]. Washington, D.C.: June 2, 
2005. 

Vocational Rehabilitation: VA Has Opportunities to Improve Services, 
but Faces Significant Challenges. [hyperlink, 
http://www.gao.gov/products/GAO-05-572T]. Washington, D.C.: April 20, 
2005. 

VA Vocational Rehabilitation and Employment Program: GAO Comments on 
Key Task Force Findings and Recommendations. [hyperlink, 
http://www.gao.gov/products/GAO-04-853]. Washington, D.C.: June 15, 
2004. 

Internal Controls Management and Evaluation Tool. [hyperlink, 
http://www.gao.gov/products/GAO-01-1008G]. Washington, D.C.: August 
2001. 

Standards for Internal Control in the Federal Government. [hyperlink, 
http://www.gao.gov/products/GAO/AIMD-00-21.3.1]. Washington, D.C.: 
November 1999. 

[End of section] 

Footnotes: 

[1] These data are based on a survey of veterans conducted in August 
2012 by the U.S. Census Bureau. 

[2] Pub. L. No. 112-56, tit II, 125 Stat. 711, 712. 

[3] See id. at § 233(d), 125 Stat. 711, 720. 

[4] VA rates veterans' disabilities that are incurred or aggravated as 
a result of their military service, which are referred to by VA as 
service-connected disabilities, from 0 percent to 100 percent in 
increments of 10 percentage points. A veteran must also have a 
military discharge that is not dishonorable to be eligible for the 
VR&E program. 

[5] 38 U.S.C. § 3102(a)(1). The law defines an employment handicap as 
an impairment of a veteran's ability to prepare for, obtain, or retain 
employment consistent with the veteran's abilities, aptitudes, and 
interests. 38 U.S.C. § 3101(1). Veterans with a 10 percent service-
connected disability may also be eligible to receive VR&E services if 
they have a serious employment handicap. 38 U.S.C. § 3102(a)(2). A 
serious employment handicap is defined as a significant impairment of 
a veteran's ability to prepare for, obtain, or retain employment 
consistent with his or her abilities, aptitudes and interests. 38 
U.S.C. § 3101(7). The definitions of employment handicap and serious 
employment handicap provide that the handicap must result in 
substantial part from a service-connected disability. 

[6] Active duty servicemembers may also be eligible for VR&E services 
under certain circumstances. For example, servicemembers may qualify 
if, while awaiting discharge from the military, they are hospitalized 
or receiving outpatient treatment at a VA facility or a facility that 
is under contract with the VA, have a disability that VA will likely 
find to be service-connected with a 20 percent or higher rating, and 
are determined to be in need of rehabilitation because of an 
employment handicap. 38 U.S.C. § 3102(a)(1)(A)(ii) and (B). Until 
December 31, 2014, servicemembers are entitled to certain VR&E 
services if they are seriously ill or injured. Pub. L. No. 110-181, 
§1631(b)(2), 122 Stat. 3, 458 (2008) (10 U.S.C 1071 note) as amended 
by Pub. L. No. 112-56, § 231, 125 Stat. 711, 719 (2011). 

[7] 38 U.S.C. § 3105(b)(1). Eligible veterans for whom a vocational 
goal is not currently considered reasonably feasible may receive 
assistance intended to help them achieve independent living. 38 U.S.C. 
§ 3109. 

[8] The 48-month limit on training services can be extended if, for 
example, the veteran has a serious employment handicap and the 
extension is necessary to accomplish the veteran's vocational or 
independent living goal. 38 U.S.C. § 3105(c)(2) and 38 C.F.R. § 
21.78(c). The 12-year limit on the eligibility period can also be 
extended for veterans who have a serious employment handicap and have 
not previously been rehabilitated to the point of employability. 38 
U.S.C. § 3103(c)(1) and 38 C.F.R. § 21.44(a)(1). 

[9] 38 C.F.R. § 21.284. Veterans who meet these conditions may receive 
such additional services even if they have previously exhausted their 
48-month entitlement to VR&E services. 38 U.S.C. § 3105(c)(1). 
Veterans who meet these conditions and have a serious employment 
handicap may receive such additional services even if they are no 
longer within their 12-year period of eligibility. 38 U.S.C. § 
3103(c)(2). 

[10] Codified at 38 U.S.C. § 3102(b). Such veterans may receive these 
additional services even if they have previously exhausted their 48 
months of services or are no longer within their 12-year eligibility 
period. 38 U.S.C. §§ 3105(b)(2) and 3103(e). 

[11] These caseload totals include veterans who are pursuing a 
rehabilitation plan to achieve either employment or independent living. 

[12] For purposes of veterans' benefits, including VR&E benefits and 
services, a veteran is defined as a person who was an active member of 
the U.S. military and was discharged or released from service under 
conditions other than dishonorable. 38 U.S.C. § 101(2). 

[13] For veterans found not entitled to VR&E program services, VRCs 
will inform them of their appeal rights and use the information 
collected during the application process to direct the veteran to 
other services, such as those offered by state workforce systems. 

[14] If a veteran is entitled to VR&E services, but his or her 
handicap is such that employment is deemed not currently feasible, and 
the veteran has a serious employment handicap resulting in substantial 
part from a service-connected disability rated at 20 percent or more, 
the veteran is entitled to independent living services and assistance 
and will receive assistance to build and pursue a plan to achieve the 
maximum independence in daily living possible. 38 U.S.C. § 3109. 
Independent living services may include counseling, diagnostic, 
medical, social, psychological, and educational services. With these 
services and benefits, some veteran may initially achieve the outcome 
of maximum independence in daily living, and, if appropriate, may work 
with VR&E to develop an employment plan and pursue suitable 
employment. For information on the independent living track, see GAO, 
VA Vocational Rehabilitation and Employment Program: Improved 
Oversight of Independent Living Services and Supports Is Needed, 
[hyperlink, http://www.gao.gov/products/GAO-13-474] (Washington, D.C.: 
June 7, 2013). 

[15] VA considers suitable employment to be work consistent with the 
veteran's abilities, aptitudes, and interests. 38 C.F.R. § 21.283(b). 

[16] VA's regulations state that the purpose of assignment to 
interrupted status is to assure that all appropriate actions have been 
taken to help a veteran continue in his or her program before 
discontinuing benefits and services. 38 C.F.R. § 21.197(c)(4). 

[17] Before the veteran's reentry to the program, VR&E must also 
determine that the reason for the discontinuance has been removed. 38 
C.F.R. 21.198(c). 

[18] Included in the Veterans Benefits Administration's on-line ASPIRE 
Dashboard, [hyperlink, http://www.vba.va.gov/reports/aspiremap.asp]. 

[19] We focus on veterans who applied in fiscal year 2003 because that 
is the earliest year for which reliable data are available. It is 
possible that tracking veterans' outcomes over a longer period of time 
would show a higher percentage of veterans placed in suitable jobs. 
Some veterans who are still participating in services 10 years after 
application may ultimately achieve success, and even some who have 
discontinued may re-apply and achieve success. 

[20] In our analysis of VA data, we found that many fiscal year 2003 
applicants did not receive employment-related services because, for 
example, they entered the independent living track instead or received 
an evaluation but never started a specific service track. 

[21] While a success rate of about 50 percent appears considerably 
lower than the 74 percent employment rehabilitation rate VA reported 
for fiscal year 2012, this discrepancy is due at least in part to how 
VA calculates its rate. VA divides the number of veterans placed in 
suitable jobs during a year by the total number placed in suitable 
jobs plus those discontinued from an employment-related rehabilitation 
plan. However, VA excludes from this calculation those still 
participating in services and those who discontinued but achieved 
certain types of MRG. If we used the same formula to calculate the 
success rate for veterans who entered the program in fiscal year 2003, 
then our rate would be 67 percent--less than but much closer to what 
VA reports. 

[22] Based on our assessment of the reliability of VA administrative 
data, we determined there may be some inaccuracies in data on the 
number of times veterans interrupted their programs. Therefore, we 
provide estimates rather than exact numbers when presenting these 
data. For more information on our data reliability assessment see 
appendix I. 

[23] Under the VOW Act provision, veterans may receive only up to 12 
additional months of benefits. By contrast, there is no specific limit 
on the number of additional months of benefits available to veterans 
who qualify under the pre-existing criteria. 

[24] Administrative data indicate that VR&E participants who received 
some employment services most commonly discontinued after failing to 
complete a training program, but also did so after receiving 
additional evaluation and planning services or during the job 
placement phase. In conducting this analysis we looked at either the 
veteran's case status immediately preceding their final 
discontinuation or, in cases where veterans interrupted services 
before finally discontinuing, the case status immediately preceding 
this interruption. 

[25] VA's Voice of the Veteran survey measures veterans' satisfaction 
with benefits and services received, including through the VR&E 
program. The results reported here are from the pilot survey fielded 
in October 2012 to January 2013; VA plans to field the survey on a 
regular, ongoing basis in the future. More specifically, the results 
on reasons for withdrawing or interrupting from the program are based 
on the responses of almost 200 veterans who had withdrawn or 
interrupted after spending at least 60 days in the program. We 
assessed the methodology of the pilot survey and concluded that the 
results we are reporting are sufficiently reliable for our purposes. 
However, these results reflect the opinions of survey respondents and 
cannot be generalized to the broader population of VR&E participants. 
For more information see appendix I. 

[26] We chose a period of 8 years for this analysis because, as shown 
in figure 4, many veterans take that long to achieve a successful 
outcome. This period of analysis provided enough time to observe 
successful outcomes and included results for veterans who applied in 
both fiscal years 2003 and 2004. We also analyzed the associations 
between specific disabilities and the likelihood and rate of achieving 
program success at various points in time after application, while 
controlling for other factors that may affect outcomes. Even after 
controlling for these factors, we still found that having a mental 
health condition was associated with a smaller chance of success. For 
more details on our analysis and results, see appendix II. 

[27] Based on our assessment of the reliability of VA administrative 
data, we determined that there may be some inaccurate data regarding 
the number of times VR&E participants received evaluation and planning 
services. Therefore, we present estimates rather than exact numbers 
for these data. For more information on our assessment of VA data, see 
appendix I. 

[28] We analyzed the association between multiple counselors and the 
likelihood and rate of achieving program success at various points in 
time following application, while controlling for other factors that 
may affect outcomes. Among other things we controlled for the number 
of times veterans discontinued from and re-entered the VR&E program, 
which could be associated with the number of counselors they worked 
with. Even after controlling for these factors, we still found that 
working with multiple counselors was associated with a smaller chance 
of success. For more details on our analysis and results, see appendix 
II. 

[29] We estimated the percentage suitably employed at several time 
points across the 56 offices. For this estimate we used the last 
office a veteran was active in since it may be most closely connected 
to the last veteran status. This estimate takes into account the 
several different possible outcomes (placed in suitable employment, 
discontinued, still participating, and achieved independent living 
goal) in estimating the probability of achieving suitable employment. 

[30] Because we analyzed outcomes for veterans who entered and exited 
the program at different points in time over a 10 -year period, and 
local economic conditions varied over this time, it was not practical 
to include factors such as state or local unemployment rates in our 
analysis. In addition, measuring demand for labor across veterans' 
diverse skills and local economies would require tailored measures 
that we were unable to obtain. 

[31] The rehabilitation rate is the percentage of successful 
rehabilitations, out of the total of successful rehabilitations plus 
cases discontinued as unsuccessful. It excludes cases where veterans 
were not successfully rehabilitated but had maximum rehabilitation 
gain, and where veterans transferred to Post-9/11 GI Bill benefits. VA 
calculates an overall rehabilitation rate for all VR&E participants 
and separate rates for different sub-groups. For the purpose of this 
section of the report, we focused on the rate for participants who 
received employment-related services. 

[32] VA can determine that the veteran (1) is employed with VR&E 
assistance, but not in a suitable job; (2) is employable, with VR&E 
assistance, but not employed; or (3) is not employable, but received 
assistance in making gains in self-management, self-advocacy, and 
independence in daily living. 

[33] However, VA officials said they may take local economic 
conditions into account in setting each office's target number of 
veterans rehabilitated. 

[34] VA created VR&E manager, VRCs, ECs and support staff workgroups. 

[35] VA already uses a similar measure in individual VRC performance 
standards: the number of veterans successfully rehabilitated, not 
including MRG cases. 

[36] For example, see GAO, Standards for Internal Control in the 
Federal Government, [hyperlink, 
http://www.gao.gov/products/GAO/AIMD-00-21.3.1] (Washington, D.C.: 
November 1999) and GAO, Managing for Results: Analytic Challenges in 
Measuring Performance, [hyperlink, 
http://www.gao.gov/products/GAO/HEHS/GGD-97-138] (Washington, D.C.: 
May 30, 1997). 

[37] Some cases are required to be tracked longer. Veterans in the 
self-employment track must be followed up on for at least a year. 

[38] Timeframes of fewer than 60 days were reported in 10 percent of 
the cases. Also, in about 8 percent of the approximately 38,000 cases, 
veterans may have obtained employment before completing training, 
which may lead to longer periods between job placement and case 
closure. 

[39] For an employment program participant, Labor defines employment 
retention as having a job in both the second and third calendar 
quarters after exiting the program. The measure is the proportion of 
such participants, out of all participants who were employed in the 
first quarter after exiting the program. 

[40] VA included a placeholder measure of veterans' satisfaction with 
the VR&E program in its fiscal year 2012 Performance and 
Accountability Report, and noted that it would establish baseline 
performance or goals based on a veterans' survey. At the time of our 
review, VR&E had not decided whether to use Voice of the Veteran 
survey data for this purpose. 

[41] VA contracted the services of J.D. Power and Associates to 
develop and pilot the Voice of the Veteran survey. The total cost of 
VA's pilot survey effort was about $977,192. VA plans to administer 
the Voice of the Veteran survey on an on-going basis, and officials 
anticipate the total contract cost to be about $1,462,500. 

[42] For generally recognized survey research practices, see Office of 
Management and Budget, Standards and Guidelines for Statistical 
Surveys, September 2006; The American Association for Public Opinion 
Research, 2011 Standard Definitions: Final Dispositions of Case Codes 
and Outcome Rates for Surveys 7th edition, AAPOR; and the American 
Statistical Association, What is a Survey, 2nd edition, 2004, 
[hyperlink, https://www.whatisasurvey.info/]. For more information on 
our assessment of the methodology and pilot results of the Voice of 
the Veteran surveys administered, see appendix I. 

[43] Nonresponse occurs when members of the sample do not participate 
in the survey at all (unit nonresponse), or do not answer particular 
questions (item nonresponse). Nonresponse may result in nonresponse 
bias--systematic errors that result in under-or overestimation of a 
true value in survey results--because nonrespondents may have provided 
substantially different answers than those who did respond. A 
nonresponse analysis is an established survey research practice used 
to determine whether nonresponse bias has occurred. It can be 
performed using a variety of methods--for example, by randomly 
selecting a sample of survey nonrespondents, interviewing them to 
obtain missing answers to key survey questions, or imputing those 
answers from administrative records or other known. 

[44] Research in survey methods suggests that--while response rates 
may not be highly correlated with the presence of nonresponse bias and 
are not considered, by themselves, to be sufficient indicators of the 
quality of survey results--response rates are a useful indicator of 
the risk of nonresponse bias. According to OMB guidelines, a survey's 
response rate is a valuable data quality and performance indicator, 
and is associated with the generalizability of a survey's results. See 
Office of Management and Budget, Guidance on Agency Survey and 
Statistical Information Collections, (Washington, D.C.: January 20, 
2006). 

[45] Specifically, VA's contract stated that the survey administrator 
would monitor the response rates of the surveys and provide reports 
that would "include recommendations (where necessary) as to whether or 
not a nonresponse bias analysis should be conducted to ascertain 
reasons or cause for low response rates." 

[46] Three pilot surveys were administered to different sub-groups of 
VR&E participants. For more information on the surveys and their 
response rates, see appendix I. 

[47] U.S. Department of Veterans Affairs, Vocational Rehabilitation 
and Employment Service, Work Measurement Study, Final Report 
(Washington, D.C.: April 21, 2011). The study was conducted for VA by 
the Millennium Corp. and New Editions Consulting, Inc. 

[48] According to VA, White River Junction had only one VRC, although 
another was added after March 31, 2013. In commenting on a draft of 
this report, VA stated that it expects another vacancy to be filled in 
January 2014, and White River Junction's average to drop to about 160 
cases per VRC. 

[49] In addition to Fargo, the offices in Wichita, Kansas; Togus, 
Maine; Lincoln, Nebraska; Pittsburgh, Pennsylvania; Huntington, West 
Virginia; and Manila, the Philippines averaged fewer than 100 cases 
per VRC. 

[50] In addition to White River Junction, the offices in Los Angeles, 
California; Hartford, Connecticut; Honolulu, Hawaii; Indianapolis, 
Indiana; Newark, New Jersey; Albuquerque, New Mexico; and Cleveland, 
Ohio averaged more than 175 cases per VRC. The Cleveland office had 
the next-highest ratio, with 206 cases per VRC. 

[51] VRCs at military treatment facilities provide VR&E services to 
wounded, ill and injured servicemembers who are in the Integrated 
Disability Evaluation System. Through the VetSuccess on Campus 
program, VRCs at college campuses provide support to student veterans 
using VA education benefits, such as career and academic counseling 
and medical and mental health referrals. 

[52] Before applying the model, VA allocates FTE positions to support 
the Integrated Disability Evaluation System (at military 
installations) and the VetSuccess on Campus program (at college 
campuses). 

[53] Workload factors include numbers of applications and cases in job 
ready status. Outcome performance factors include numbers of 
entitlement determinations and rehabilitations. Accuracy performance 
factors include the accuracy of entitlement determinations and program 
outcomes (rehabilitation or unsuccessful discontinuance). 

[54] In addition to VR&E services, VR&E can provide free educational 
and vocational counseling to certain veterans and their dependents, 
and servicemembers. These include veterans eligible for VA education 
and VR&E benefits, and servicemembers within 180 days of expected 
discharge. Services include interest and aptitude testing; assistance 
in setting occupational goals; and assistance in identifying education 
and training institutions that can be used to achieve occupational 
goals. 

[55] In previous work, we presented research that identified drawbacks 
of basing program funding on performance measures. GAO, Vocational 
Rehabilitation Funding Formula: Options for Improving Equity in State 
Grants and Considerations for Performance Incentives, [hyperlink, 
http://www.gao.gov/products/GAO-09-798] (Washington, D.C.: September 
30, 2009). 

[56] GAO, DOJ Workforce Planning: Grant-Making Components Should 
Enhance the Utility of Their Staffing Models, [hyperlink, 
http://www.gao.gov/products/GAO-13-92] (Washington, D.C.: December 14, 
2012). 

[57] In some cases veterans may work with multiple VRCs even if an 
office does not take a specialization approach. For example, if a VRC 
retires, that VRC's cases would be redistributed among other VRCs. 

[58] See [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1]. 

[59] GAO, Human Capital: A Guide for Assessing Strategic Training and 
Development Efforts in the Federal Government, [hyperlink, 
http://www.gao.gov/products/GAO-04-546G] (Washington, D.C.: March 
2004). 

[60] For the VBA-sponsored study, see General Dynamics, Skills Survey 
Results Report for Vocational Rehabilitation and Employment (VR&E) 
Services (Orlando, FL: June 7, 2010). 

[61] GAO, VA Vocational Rehabilitation and Employment: Better 
Incentives, Workforce Planning, and Performance Reporting Could 
Improve Program, [hyperlink, http://www.gao.gov/products/GAO-09-34] 
(Washington, D.C.: January 26, 2009). 

[62] Tax incentives, such as the Work Opportunity Tax Credit and the 
disabled access credit, are available to employers to encourage the 
hiring, retention, and accommodation of workers with disabilities. 

[63] For the VR&E-sponsored study, see ICF International, VR&E Service 
Employment of Individuals with Severe Injuries Study: Final Report 
(Fairfax, VA: June 13, 2012). 

[64] The "WINRS" part of the acronym represents the first five 
regional offices that tested the original system. These offices 
include Waco, Texas; Indianapolis, Indiana; Newark, New Jersey; 
Roanoke, Virginia; and Seattle, Washington. 

[65] In addition to CWINRS data, we also analyzed data on the specific 
type of training veterans received from VA's Benefits Delivery Network 
and data on regional office staffing from VA's ProClarity system. 

[66] The discontinued category also includes veterans who started 
employment-related services but were later found not eligible. 

[67] VA renamed the Washington, D.C., regional office as the National 
Capital Region Benefits Office in November 2012. For purposes of our 
review, we refer to the National Capital Region Benefits Office as a 
regional office. 

[68] The Employment Rehabilitation Rate differs from VR&E's general 
Rehabilitation Rate in that it does not include veterans placed in the 
Independent Living Program. 

[69] We interviewed the VR&E Officer at the Boston, Massachusetts 
regional office, who manages VR&E operations at the Manchester, New 
Hampshire regional office. Also, Manchester did not have an EC at the 
time of our review. 

[70] See Office of Management and Budget, Standards and Guidelines for 
Statistical Surveys (September 2006). 

[71] See the American Association for Public Opinion Research, 2011 
Standard Definitions: Final Dispositions of Case Codes and Outcome 
Rates for Surveys 7th edition; and the American Statistical 
Association, What is a Survey, 2nd edition (2004), [hyperlink, 
https://www.whatisasurvey.info/]. 

[72] For additional information on survival modeling methods, please 
see Paul D. Allison, Survival Analysis Using SAS®: A Practical Guide, 
2d ed. (Cary, NC: SAS Institute, Inc., 2010). 

[73] Allison, 175-176. 

[74] The 95 percent confidence interval of this hazard ratio ranged 
from 0.72 to 0.75. 

[75] The 95 percent confidence interval of this hazard ratio ranges 
from 1.4 to 1.6. 

[End of section] 

GAO’s Mission: 

The Government Accountability Office, the audit, evaluation, and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the 
performance and accountability of the federal government for the 
American people. GAO examines the use of public funds; evaluates 
federal programs and policies; and provides analyses, recommendations, 
and other assistance to help Congress make informed oversight, policy, 
and funding decisions. GAO’s commitment to good government is 
reflected in its core values of accountability, integrity, and 
reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO’s website [hyperlink, http://www.gao.gov]. Each 
weekday afternoon, GAO posts on its website newly released reports, 
testimony, and correspondence. To have GAO e-mail you a list of newly 
posted products, go to [hyperlink, http://www.gao.gov] and select 
“E-mail Updates.” 

Order by Phone: 

The price of each GAO publication reflects GAO’s actual cost of 
production and distribution and depends on the number of pages in the 
publication and whether the publication is printed in color or black 
and white. Pricing and ordering information is posted on GAO’s 
website, [hyperlink, http://www.gao.gov/ordering.htm]. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or 
TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card, 
MasterCard, Visa, check, or money order. Call for additional 
information. 

Connect with GAO: 

Connect with GAO on facebook, flickr, twitter, and YouTube.
Subscribe to our RSS Feeds or E mail Updates. Listen to our Podcasts.
Visit GAO on the web at [hyperlink, http://www.gao.gov]. 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 
Website: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]; 
E-mail: fraudnet@gao.gov; 
Automated answering system: (800) 424-5454 or (202) 512-7470. 

Congressional Relations: 

Katherine Siggerud, Managing Director, siggerudk@gao.gov: 
(202) 512-4400: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7125: 
Washington, DC 20548. 

Public Affairs: 

Chuck Young, Managing Director, youngc1@gao.gov: 
(202) 512-4800: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7149: 
Washington, DC 20548. 

[End of document]