This is the accessible text file for GAO report number GAO-13-130 
entitled 'VA Health Care: Reliability of Reported Outpatient Medical 
Appointment Wait Times and Scheduling Oversight Need Improvement' 
which was released on January 18, 2012. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as 
part of a longer term project to improve GAO products' accessibility. 
Every attempt has been made to maintain the structural and data 
integrity of the original printed product. Accessibility features, 
such as text descriptions of tables, consecutively numbered footnotes 
placed at the end of the file, and the text of agency comment letters, 
are provided but may not exactly duplicate the presentation or format 
of the printed version. The portable document format (PDF) file is an 
exact electronic replica of the printed version. We welcome your 
feedback. Please E-mail your comments regarding the contents or 
accessibility features of this document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

United States Government Accountability Office: 
GAO: 

Report to Congressional Requesters: 

December 2012: 

VA Health Care: 

Reliability of Reported Outpatient Medical Appointment Wait Times and 
Scheduling Oversight Need Improvement: 

GAO-13-130: 

GAO Highlights: 

Highlights of GAO-13-130, a report to congressional requesters. 

Why GAO Did This Study: 

VHA provided nearly 80 million outpatient medical appointments to 
veterans in fiscal year 2011. While VHA has reported continued 
improvements in achieving access to timely medical appointments, 
patient complaints and media reports about long wait times persist. 
GAO was asked to evaluate VHA’s scheduling of timely medical 
appointments. GAO examined (1) the extent to which VHA’s approach for 
measuring and monitoring medical appointment wait times reflects how 
long veterans are waiting for appointments; (2) the extent to which 
VAMCs are implementing VHA’s policies and processes for appointment 
scheduling, and any problems encountered in ensuring veterans’ access 
to timely medical appointments; and (3) VHA’s initiatives to improve 
veterans’ access to medical appointments. To conduct this work, GAO 
made site visits to 23 clinics at four VAMCs, the latter selected for 
variation in size, complexity, and location. GAO also reviewed VHA’s 
policies and data, and interviewed VHA officials. 

What GAO Found: 

Outpatient medical appointment wait times reported by the Veterans 
Health Administration (VHA), within the Department of Veterans Affairs 
(VA), are unreliable. Wait times for outpatient medical appointments—-
referred to as medical appointments-—are calculated as the number of 
days elapsed from the desired date, which is defined as the date on 
which the patient or health care provider wants the patient to be 
seen. The reliability of reported wait time performance measures is 
dependent on the consistency with which schedulers record the desired 
date in the scheduling system in accordance with VHA’s scheduling 
policy. However, VHA’s scheduling policy and training documents for 
recording desired date are unclear and do not ensure consistent use of 
the desired date. Some schedulers at Veterans Affairs medical centers 
(VAMC) that GAO visited did not record the desired date correctly. For 
example, three schedulers changed the desired date based on 
appointment availability; this would have resulted in a reported wait 
time that was shorter than the patient actually experienced. VHA 
officials acknowledged limitations of measuring wait times based on 
desired date, and described additional information used to monitor 
veterans’ access to medical appointments, including patient 
satisfaction survey results. Without reliable measurement of how long 
patients are waiting for medical appointments, however, VHA is less 
equipped to identify areas that need improvement and mitigate problems 
that contribute to wait times. 

While visiting VAMCs, GAO also found inconsistent implementation of 
VHA’s scheduling policy that impedes VAMCs from scheduling timely 
medical appointments. For example, four clinics across three VAMCs did 
not use the electronic wait list to track new patients that needed 
medical appointments as required by VHA scheduling policy, putting 
these clinics at risk for losing track of these patients. Furthermore, 
VAMCs’ oversight of compliance with VHA’s scheduling policy, such as 
ensuring the completion of required scheduler training, was 
inconsistent across facilities. VAMCs also described other problems 
with scheduling timely medical appointments, including VHA’s outdated 
and inefficient scheduling system, gaps in scheduler and provider 
staffing, and issues with telephone access. For example, officials at 
all VAMCs GAO visited reported that high call volumes and a lack of 
staff dedicated to answering the telephones impede scheduling of 
timely medical appointments. In January 2012, VHA distributed 
telephone access best practices that, if implemented, could help 
improve telephone access to clinical care. 

VHA is implementing a number of initiatives to improve veterans’ 
access to medical appointments such as expanded use of technology to 
interact with patients and provide care, which includes the use of 
secure messaging between patients and their health care providers. VHA 
also is piloting a new initiative to provide health care services 
through contracts with community providers that aims to reduce travel 
and wait times for veterans who are unable to receive certain types of 
care within VHA in a timely way. 

What GAO Recommends: 

GAO recommends that VHA take actions to (1) improve the reliability of 
its medical appointment wait time measures, (2) ensure VAMCs 
consistently implement VHA’s scheduling policy, (3) require VAMCs to 
allocate staffing resources based on scheduling needs, and (4) ensure 
that VAMCs provide oversight of telephone access and implement best 
practices to improve telephone access for clinical care. VA concurred 
with GAO’s recommendations. 

View [hyperlink, http://www.gao.gov/products/GAO-13-130]. For more 
information, contact Debra Draper at (202) 512-7114 or draperd@gao.gov. 

[End of section] 

Contents: 

Letter: 

Background: 

Reported Medical Appointment Wait Times Are Unreliable; VHA Uses 
Additional Methods to Monitor Patients' Access to Medical Appointments: 

Inconsistent Implementation of VHA's Scheduling Policy and Other 
Problems Impede VAMCs' Ability to Schedule Timely Medical Appointments: 

VHA Is Implementing a Number of Initiatives to Improve Access to 
Medical Appointments: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments and Our Evaluation: 

Appendix I: Comments from the Department of Veterans Affairs: 

Appendix II: GAO Contact and Staff Acknowledgments: 

Related GAO Products: 

Tables: 

Table 1: Selected Fiscal Year 2012 VHA Medical Appointment Wait Time 
Performance Measures: 

Table 2: Number of Schedulers at Each VAMC Visited Who Incorrectly 
Recorded the Medical Appointment Desired Date, by Error Type: 

Table 3: Number of Clinics That Did Not Implement Selected Elements of 
the VHA's Scheduling Policy, by VAMC Visited: 

Abbreviations: 

CBOC: community-based outpatient clinic: 

FDPP: Facility Director Performance Plan: 

NDPP: Network Director Performance Plan: 

OIG: Office of Inspector General: 

PACT: Patient-Aligned Care Team: 

PAR: Performance and Accountability Report: 

Project ARCH: Access Reached Closer to Home: 

VA: Department of Veterans Affairs: 

VAMC: VA Medical Center: 

VHA: Veterans Health Administration: 

VISN: Veterans Integrated Service Network: 

VistA: Veterans Health Information Systems and Technology Architecture: 

[End of section] 

United States Government Accountability Office: 
Washington, DC 20548: 

December 21, 2012: 

Congressional Requesters: 

The Veterans Health Administration (VHA), within the Department of 
Veterans Affairs (VA), operates one of the nation's largest health 
care systems. Its medical facilities include 152 VA medical centers 
(VAMC), as well as more than 800 community-based outpatient clinics 
(CBOC) and ambulatory care centers that operate under supervision of 
the VAMCs.[Footnote 1] At these facilities, VHA provides outpatient 
care through primary and specialty care clinics.[Footnote 2] In fiscal 
year 2011, there were more than 8 million veterans enrolled in VHA's 
health system, and VHA provided nearly 80 million outpatient medical 
appointments to veterans for primary and specialty care.[Footnote 3] 

Although access to timely medical appointments is critical to ensuring 
that veterans obtain needed medical care, long wait times and 
inadequate scheduling processes at VHA medical facilities have been 
long-standing problems. For example, in 2001, GAO reviewed the 
timeliness of medical appointments and found that two-thirds of the 
specialty care clinics visited had wait times longer than 30 days, 
although some clinics had made progress in reducing wait times, 
primarily by improving their scheduling processes and making better 
use of their staff.[Footnote 4] Later, in 2007, the VA Office of 
Inspector General (OIG) reported that VHA facilities did not always 
follow VHA's scheduling policies and processes and that the accuracy 
of VHA's reported wait times for medical appointments was unreliable. 
[Footnote 5] Most recently, in 2012, the VA OIG reported that VHA was 
not providing all new veterans with timely access to full mental 
health evaluations, and had overstated its success in providing 
veterans with timely new and follow-up appointments for mental health 
treatment.[Footnote 6] 

VHA has reported continued improvements in measuring and achieving 
timely access to medical appointments. For example, in fiscal year 
2011, VHA had a goal of scheduling medical appointments within 14 days 
of the patient's or provider's desired medical appointment date, 
[Footnote 7] and in that year, VA reported that it completed 95 
percent of specialty care medical appointments and 94 percent of 
primary care medical appointments within this time frame. However, 
patient complaints and media reports about long wait times have 
persisted, prompting renewed concerns about excessive medical 
appointment wait times. You asked us to evaluate VHA's scheduling of 
timely medical appointments. We examined (1) the extent to which VHA's 
approach for measuring and monitoring medical appointment wait times 
reflects how long veterans are waiting; (2) the extent to which VAMCs 
are implementing VHA's policies and processes for medical appointment 
scheduling, and any problems encountered in ensuring veterans' access 
to timely medical appointments as identified by VAMCs; and (3) VHA's 
initiatives to improve veterans' access to medical appointments. 

To address all three objectives, we interviewed VHA central office 
officials responsible for medical appointment scheduling policy, 
medical appointment wait time measurement, and initiatives to improve 
access to timely medical appointments. We also conducted site visits 
to four VAMCs selected for variation in size and complexity, 
geographic location, or role as a pilot site for a VHA initiative to 
improve access to timely medical appointments.[Footnote 8] These four 
VAMCs were located in Dayton, Ohio; Fort Harrison, Montana; Los 
Angeles, California; and Washington, D.C. At each site, we visited the 
VAMC as well as the affiliated CBOCs that had among the highest volume 
of medical appointments.[Footnote 9] We also visited the highest 
volume ambulatory care center at one VAMC that had such facilities. 
[Footnote 10] At each VAMC, CBOC, and ambulatory care center, we 
visited an outpatient primary care clinic, and where available, one or 
more outpatient specialty care clinics for a total of 23 clinics--9 
primary care and 14 specialty care clinics--the clinics were among 
those with the highest medical appointment volume.[Footnote 11] 
Results from our site visits cannot be generalized to other VAMCs. We 
also interviewed the directors and relevant staff of the four Veterans 
Integrated Service Networks (VISN), or regional networks of care, for 
the sites we visited.[Footnote 12] 

To examine the extent to which VHA's approach for measuring and 
monitoring medical appointment wait times reflects how long veterans 
are waiting, we reviewed VHA's outpatient medical appointment 
scheduling policy and processes and training documents based on the 
policy, as well as documents related to performance accountability 
that include wait time measures.[Footnote 13] At each of the four 
VAMCs we visited, we interviewed the leadership team, scheduling 
managers, and managers from all of our selected clinics about 
oversight activities to ensure the accuracy of scheduling data, and 
about what data and measures they use to manage clinics and improve 
medical appointment timeliness. In addition, we interviewed schedulers 
from the 9 primary care clinics and 10 of the 14 specialty care 
clinics we visited using a structured protocol to determine the 
accuracy with which schedulers determined and recorded medical 
appointment data for hypothetical patient medical appointments into 
the scheduling system in accordance with VHA's scheduling policy. 
[Footnote 14] We also compared reported medical appointment wait times 
for the clinics we visited to information gathered during our site 
visits.[Footnote 15] 

To determine the extent to which VAMCs are implementing VHA's 
scheduling policy and processes, and to gather information on problems 
encountered in scheduling timely medical appointments, we reviewed 
VHA's scheduling policy, interviewed VHA central office officials 
responsible for the scheduling policy, and obtained information about 
scheduling practices from officials at each of the four VAMCs we 
visited. Specifically, for each of the VAMCs, we interviewed 
leadership, scheduling managers, clinic managers, patient advocates, 
and case managers. We obtained information about the VAMCs' oversight 
to ensure compliance with policy, and about problems staff at these 
facilities say they experience in scheduling timely medical 
appointments. We reviewed the implementation of selected elements of 
VHA's scheduling policy at both the VAMC and individual clinic level. 
These elements included the use of VHA's scheduling software to 
schedule medical appointments, the use of the electronic wait 
list[Footnote 16] for tracking patients new to a clinic that are 
waiting for medical appointments, and the use of the recall/reminder 
software to facilitate reminders for patients that need to return to 
the clinic for follow-up medical appointments. We also obtained from 
VHA and reviewed VAMCs' fiscal year 2011 certifications of compliance 
with VHA's scheduling policy--a required annual self-certification--as 
well as documentation of scheduler training completion obtained from 
the four VAMCs visited. Finally, we reviewed patient complaints about 
telephone responsiveness collected by each VAMC's Office of the 
Patient Advocate. 

To examine VHA's initiatives to improve veterans' access to timely 
medical appointments, we interviewed VHA central office officials to 
obtain information about selected initiatives and reviewed relevant 
VHA documents outlining these initiatives. We also interviewed 
officials at the VAMCs we visited about the implementation of these 
initiatives and officials at the Billings Clinic, a non-VA health care 
facility involved in the implementation of one of the initiatives. 
[Footnote 17] 

We conducted this performance audit from February 2012 to December 
2012 in accordance with generally accepted government auditing 
standards. Those standards require that we plan and perform the audit 
to obtain sufficient, appropriate evidence to provide a reasonable 
basis for our findings and conclusions based on our audit objectives. 
We believe that the evidence obtained provides a reasonable basis for 
our findings and conclusions based on our audit objectives. 

Background: 

VHA's health care system is geographically divided into 21 VISNs, each 
of which is headed by a VISN director. Each VISN is comprised of a 
network of VAMCs, and the VISN office serves as the basic budgetary 
and decision-making unit for providing health care services to 
veterans within that geographical area. Each VAMC and its affiliated 
CBOCs and ambulatory care centers are headed by a VAMC director who 
manages administrative functions, and a chief of staff who manages 
clinical functions for these facilities. VHA's Central Office 
establishes system-wide scheduling policy. 

VHA Medical Appointment Scheduling Policy: 

VHA's scheduling policy establishes processes and procedures for 
scheduling medical appointments, and for ensuring the competency of 
staff directly or indirectly involved in the scheduling process. This 
policy is designed to help VAMCs meet VHA's commitment to scheduling 
medical appointments with no undue waits or delays. Specifically, 
VHA's scheduling policy includes, but is not limited to, the following 
requirements: 

* Requires VAMCs to use VHA's Veterans Health Information Systems and 
Technology Architecture (VistA) medical appointment scheduling system 
to schedule medical appointments.[Footnote 18] 

* Requires VAMCs to keep appointment schedules open and available for 
patients to make medical appointments at least 3 to 4 months into the 
future. 

* Requires schedulers to record in the VistA scheduling system the 
date on which the patient or provider wants the patient to be seen as 
the desired date. To determine the desired date, schedulers should be 
in communication with the patient when scheduling the medical 
appointment. 

* Requires schedulers to record the desired date correctly and 
describes how to determine and record the desired date for new 
patients--patients who haven't been seen by a health care provider in 
a clinic within the past 2 years, including those scheduled in 
response to a consult request--as well as specifying how to determine 
the desired date for established patients' follow-up medical 
appointments--patients who have been seen within the past two years. 
[Footnote 19] 

* Requires VAMCs to track new patients waiting for medical 
appointments using the electronic wait list within VistA and to remind 
established patients of follow-up medical appointments using the 
recall/reminder software within VistA, which enables clinics to create 
a list of established patients who need follow-up medical appointments 
more than 3 or 4 months in the future. 

Additionally, VHA has a separate directive that establishes policy on 
the provision of telephone service related to clinical care, including 
facilitating telephone access for medical appointment management. 
[Footnote 20] 

Officials at the VHA central office, VISN, and VAMC all have oversight 
responsibilities for the implementation of VHA's scheduling policy. In 
the VHA central office, the Director of Systems Redesign, through the 
Office of the Deputy Undersecretary for Health for Operations and 
Management, is responsible for the oversight and implementation of 
medical appointment scheduling requirements. This oversight includes 
measurement and monitoring of ongoing performance. Each VISN director, 
or designee, is responsible for oversight of enrollment and medical 
appointment scheduling for eligible veterans.[Footnote 21] Each VAMC 
director, or designee, is responsible for ensuring that clinics' 
scheduling of medical appointments complies with VHA's scheduling 
policy, including clinics in affiliated CBOCs and ambulatory care 
centers. In addition, the VAMC director is responsible for ensuring 
that any staff who can schedule medical appointments in the VistA 
scheduling system has completed VHA scheduler training.[Footnote 22] 

Starting in fiscal year 2007, VHA required every VAMC to annually self-
certify compliance with VHA's scheduling policy.[Footnote 23] This 
certification is signed by the VAMC director and also encompasses 
scheduling compliance in affiliated CBOCs and ambulatory care centers. 
For fiscal year 2011, the certification required VAMCs to self-certify 
compliance, partial compliance, or noncompliance with more than 30 
individual aspects of VHA's scheduling policy as well as overall 
compliance, partial compliance, or noncompliance with VHA's scheduling 
policy as a whole. According to officials, VHA's central office does 
not penalize noncompliance with the certification and expects 
oversight to be managed locally. VHA's central office uses this 
certification of compliance as a tool for VAMCs to identify and 
improve performance on important aspects of the policy. 

VistA Information Technology System: 

VistA is the single integrated health information system used 
throughout VHA in all of its health care settings. There are many 
different VistA applications for clinical, administrative, and 
financial functions, including VHA's electronic medical record, known 
as the Computerized Patient Record System, and the scheduling system. 
As we reported in May 2010, the VistA scheduling system is more than 
25 years old and inefficient in facilitating care coordination between 
different sites.[Footnote 24] In 2000, VHA began an initiative to 
modernize the scheduling system, but VA terminated the project in 
2009. We also reported that VA's efforts to successfully replace the 
scheduling system were hindered by weaknesses in its project 
management processes and lack of effective oversight. 

Wait Time Measurement and Performance: 

In 1995, VHA established a goal of scheduling primary and specialty 
care medical appointments within 30 days to ensure veterans' timely 
access to care.[Footnote 25] In fiscal year 2011, VHA shortened the 
wait time goal to 14 days for both primary and specialty care medical 
appointments based on improved performance reported in previous years. 
Specifically, VA's reported wait times for fiscal year 2010 showed 
that nearly all primary care and specialty care medical appointments 
were scheduled within 30 days of desired date. In fiscal year 2012, 
VHA added a goal of completing primary care medical appointments 
within 7 days of the desired date. 

To facilitate accountability for achieving its wait time goals, VHA 
includes wait time measures--referred to as performance measures--in 
its VISN and VAMC directors' performance contracts known as Network 
Director Performance Plans (NDPP) and Facility Director Performance 
Plans (FDPP), respectively.[Footnote 26] Wait time performance 
measures also are included in VA's budget submissions and performance 
reports to Congress and stakeholders; the performance reports are 
published annually in VA's Performance and Accountability Report 
(PAR).[Footnote 27] However, the medical appointment wait time 
performance measures included in the NDPPs and FDPPs differ from the 
measures that are reported in the PAR. (See table 1.) For example, in 
fiscal year 2012, VHA's wait time goal of 7 days for primary care 
medical appointments was reflected in the NDPP and FDPP performance 
measures, but the fiscal year 2012 PAR reported primary care wait time 
performance using a 14-day standard.[Footnote 28] The performance 
measures have also changed over time.[Footnote 29] 

At the time of our review, all of VHA's medical appointment wait time 
performance measures reflected the number of days elapsed from the 
patient's or provider's desired date, which is recorded in the VistA 
scheduling system by VAMCs' schedulers. According to VHA central 
office officials, VHA measures wait times based on desired date in 
order to capture the patient's experience waiting and to reflect the 
patient's or provider's wishes; which is not reflected by other 
available wait time measures. 

Table 1: Selected Fiscal Year 2012 VHA Medical Appointment Wait Time 
Performance Measures: 

Performance plan or report: Network Director's Performance Plans and 
Facility Director's Performance Plans (NDPP, FDPP)[A]; 

Wait time performance measures: 

Percent of patients waiting for a specialty care appointment longer 
than 14 days from the desired date[B]; 

Percent of primary care appointments completed within 7 days of the 
desired date[C]; 

Same day access with primary care provider (percent of requested same 
day primary care appointments completed within one day)[C]. 

Performance plan or report: VA Budget Submission and Performance and 
Accountability Report (PAR)[D]; 

Wait time performance measures: 

Percent of new patient primary care appointments completed within 14 
days of the desired date[E]; 

Percent of established patient primary care appointments completed 
within 14 days of the desired date; 

Percent of new patient specialty care appointments completed within 14 
days of the desired date[E]; 

Percent of established patient specialty care appointments completed 
within 14 days of the desired date. 

Source: VA. 

Notes: New and established patient appointments, and primary care and 
specialty care appointments refer to outpatient medical appointments. 

[A] The NDPP and FDPP are the network (VISN) and facility (VAMC) 
directors' contracts that include performance measures against which 
directors are rated at the end of the fiscal year, and are monitored 
throughout the year. 

[B] For fiscal year 2012, the specialty care medical appointment wait 
time measure represents the percent of patients waiting for at least 
one appointment longer than 14 days from the desired date at a given 
point in time. The measure was collected twice a month and VISN and 
VAMC directors were rated against the average of these scores at the 
end of the year. The plans also included additional performance 
measures specifically for mental health appointment timeliness that 
are outside the scope of this report. 

[C] For fiscal year 2012, the plans included five individual measures 
related to primary care and a sixth measure that was a composite of 
the five individual measures. Two of the five individual primary care 
measures pertained to primary care medical appointment wait times; 
these two wait time measures represent the percent of appointments 
completed in the specified time frame. The three other individual 
measures were percentage of total encounters that occur by telephone 
(telephone utilization), percentage of primary care appointments with 
patient's assigned primary care provider (continuity of care), and 
percentage of patients discharged from hospital who were contacted by 
their primary care provider within 2 days (post-hospital discharge 
contact). At the end of the year, VISN and VAMC directors were scored 
for each of the five individual primary care measures and rated 
against the single composite measure. They were rated as meeting the 
composite measure if they met targets for at least three of the five 
individual primary care measures. 

[D] VA prepares a congressional budget justification that provides 
details supporting the policy and funding decisions in the President's 
budget request that is submitted to Congress prior to the beginning of 
each fiscal year. VA also publishes an annual PAR, which contains 
performance targets and results achieved against those targets in the 
previous year. For fiscal year 2012, the four PAR wait time measures 
represent the percent of each type of appointment completed within 14 
days of the desired date. The cumulative year-to-date scores were 
reported on the PAR. 

[E] Generally, for wait time measurement purposes, VA defines new 
patients as those who have not been seen in a particular clinic at 
that facility within the last 2 years. This includes appointments for 
newly enrolled patients as well as those scheduled in response to a 
consult request. 

[End of table] 

Reported Medical Appointment Wait Times Are Unreliable; VHA Uses 
Additional Methods to Monitor Patients' Access to Medical Appointments: 

Medical appointment wait times used for measuring and assessing 
performance toward VHA's wait time goals are unreliable due to 
problems with recording the appointment desired date in the VistA 
scheduling system. Acknowledging limitations of the wait time 
measures, VHA uses additional information to monitor patients' access 
to medical appointments. 

Reported Medical Appointment Wait Times Derived from Desired Date Are 
Unreliable: 

VHA measures its medical appointment wait times as the number of days 
that have elapsed from the patient's or provider's desired date. 
Consequently, the reliability of reported wait time performance is 
dependent on the consistency with which schedulers record the desired 
date in the VistA scheduling system. However, aspects of VHA's 
scheduling policy and training documents regarding how to determine 
and record the desired date are unclear and do not ensure replicable 
and reliable use of the desired date. In addition, we found that some 
schedulers at select VAMCs did not correctly implement other aspects 
of VHA's scheduling policy for recording the desired date. 

Unclear Desired Date Policy: 

Aspects of VHA's scheduling policy and related training documents on 
how to determine and record the desired date are unclear and do not 
ensure replicable and reliable recording of the desired date by the 
large number of staff across VHA who can schedule medical appointments 
in the VistA scheduling system.[Footnote 30] Specifically, VHA's 
scheduling policy and related scheduler training documents do not 
provide consistent guidance about when or whether the desired date 
should be based on the patient's or provider's preference. While the 
policy defines desired date as "the date on which the patient or 
provider wants the patient to be seen," it also instructs that the 
"the desired date needs to be defined by the patient" for new patient 
medical appointments, medical appointments scheduled in response to 
consult requests, and established patient follow-up medical 
appointments. When there is a conflict between the provider and 
patient desired date, the scheduler is instructed to contact the 
provider for a decision on the return time frame, but the policy and 
training documents do not clearly describe under what circumstances 
the provider's date should be used as the desired date. Further, 
providers may designate a desired appointment time frame for a follow-
up medical appointment rather than a specific date; in such cases, the 
policy is unclear as to which date within the provider's designated 
time frame the scheduler should enter as the desired date. The 
scheduling policy and training do not provide sufficient guidance to 
ensure consistent use of desired date in these various scheduling 
scenarios. 

VHA central office officials responsible for developing VHA's 
scheduling policy and related training documents told us that the 
desired date is intentionally broad to account for all of the 
scheduling scenarios that may exist. However, leadership officials 
from the four VAMCs we visited and their corresponding VISNs reported 
problems with the unclear guidance on the desired date definition, and 
difficulties achieving consistent and correct use of the desired date 
by their schedulers. In addition, given the ambiguity in the 
scheduling policy and related training documents, there are different 
interpretations of the desired date between officials at different 
levels. For example, a VISN director stated that if a provider gives a 
desired time frame, the scheduler is to use the earliest date in that 
range as the desired date; whereas a provider in a specialty care 
clinic at the VAMC we visited within that VISN stated that the clinic 
uses the latest date in the range to meet the 14-day specialty care 
medical appointment scheduling goal. 

Additionally, when presented with various scheduling scenarios, 
schedulers at the VAMCs we visited determined and recorded the desired 
date differently. For example, when posed with the question "What date 
do you enter into the scheduling system as the desired date for an 
established patient follow-up medical appointment?", 12 schedulers 
said they would enter the patient's desired date, 4 said the 
provider's date, and the remaining 3 said they used the next available 
medical appointment date. When posed with the question "If the 
patient's stated desired date conflicts with the provider's designated 
desired date or time frame, what date do you enter as the desired 
date?", 1 scheduler said that the patient's desired date would be 
entered, while another said the desired date has to come from the 
provider. The variation in schedulers' interpretation of the desired 
date suggests confusion about its correct use in different scheduling 
scenarios. 

Errors in Recording Desired Date: 

Although unclear about when to use the patient's or provider's desired 
date, VHA's scheduling policy clearly instructs that, in all 
circumstances, the desired date should be defined without regard to 
schedule capacity, and should not be altered once established to 
reflect a medical appointment date the patient accepts because of lack 
of medical appointment availability on the desired date. However, we 
found that at least one scheduler from each of the VAMCs we visited 
did not correctly implement these aspects of the policy when recording 
the desired date in the VistA scheduling system for specific 
hypothetical scheduling situations.[Footnote 31] As summarized in 
table 2, we identified the following three types of errors, each of 
which would have resulted in desired dates that did not accurately 
reflect the patients' or providers' desired date, as well as 
potentially result in the reporting of more favorable wait times for 
those medical appointments.[Footnote 32] 

* Determined appointment availability prior to establishing desired 
date: Although VHA's scheduling policy requires schedulers to 
establish the desired date for a medical appointment without regard to 
the schedule capacity, four schedulers from three VAMCs determined the 
clinic's next available medical appointment dates before establishing 
a desired date. Therefore, reported wait times for these appointments 
may not have accurately reflected how long patients actually waited. 

* Altered original desired date based on appointment availability: 
Three schedulers from two VAMCs established a desired date that was 
recorded in the VistA scheduling system independent of schedule 
capacity, but later altered the desired date because of appointment 
availability. Specifically, two of the three schedulers altered the 
originally established desired date to match the agreed-upon 
appointment date, which would have incorrectly resulted in no wait 
time reported for the appointment. The third scheduler altered the 
established desired date when there was no appointment availability 
within 2 weeks of that date; which would have resulted in an 
incorrectly reported wait time that was shorter than the patient 
actually waited from his or her original desired date. 

* Recorded a new desired date when rescheduling appointment: 
Additionally, eight schedulers from three VAMCs incorrectly recorded a 
new desired date when rescheduling an appointment canceled by the 
clinic rather than keeping the original desired date as required by 
VHA's scheduling policy. Changing the desired date in this way would 
incorrectly decrease the reported wait times for the rescheduled 
appointments; veterans actually would wait longer than the reported 
wait times indicated. 

Table 2: Number of Schedulers at Each VAMC Visited Who Incorrectly 
Recorded the Medical Appointment Desired Date, by Error Type: 

Desired date recording error: Determined medical appointment 
availability prior to establishing desired date[A]; 
Number of schedulers who demonstrated error (number of schedulers 
interviewed at each VAMC): 
VAMC A (3): 1; 
VAMC B (7): 2; 
VAMC C (4): 1; 
VAMC D (5): 0; 
Total schedulers who demonstrated error: 4. 

Desired date recording error: Altered original desired date based on 
medical appointment availability[A]; 
Number of schedulers who demonstrated error (number of schedulers 
interviewed at each VAMC): 
VAMC A (3): 0; 
VAMC B (7): 2; 
VAMC C (4): 1; 
VAMC D (5): 0; 
Total schedulers who demonstrated error: 3. 

Desired date recording error: Recorded a new desired date when 
rescheduling a medical appointment[B]; 
Number of schedulers who demonstrated error (number of schedulers 
interviewed at each VAMC): 
VAMC A (3): 0; 
VAMC B (7): 3; 
VAMC C (4): 3; 
VAMC D (5): 2; 
Total schedulers who demonstrated error: 8. 

Desired date recording error: Total number of errors; 
Number of schedulers who demonstrated error (number of schedulers 
interviewed at each VAMC): 
VAMC A (3): 1; 
VAMC B (7): 7; 
VAMC C (4): 5; 
VAMC D (5): 2; 
Total schedulers who demonstrated error: 15. 

Desired date recording error: Total number of schedulers who 
demonstrated at least one error; 
Number of schedulers who demonstrated error (number of schedulers 
interviewed at each VAMC): 
VAMC A (3): 1; 
VAMC B (7): 6; 
VAMC C (4): 3; 
VAMC D (5): 2; 
Total schedulers who demonstrated error: 12. 

Source: GAO analysis of scheduler interviews. 

Notes: This table presents the results of our interviews with 19 
schedulers from the four VAMCs visited. We used structured questions 
to test how schedulers would determine and enter the desired date for 
specific medical appointment types using hypothetical patients. We 
identified three types of desired date scheduling errors which, for 
actual patient medical appointments, would have resulted in a desired 
date that would not accurately reflect the patient or provider's 
desired date, as well as potentially result in reporting of more 
favorable wait times for those medical appointments. A scheduler could 
have demonstrated more than one error. 

[A] According to VHA's scheduling policy, when scheduling new patient 
medical appointments--including medical appointments in response to a 
consult request--and established patient follow-up medical 
appointments, the medical appointment desired date needs to be defined 
by the patient without regard to schedule capacity. Once the patient's 
desired date has been established, it must not be altered to reflect a 
medical appointment date patients agree to accept due to lack of 
medical appointment availability on the original desired date. 

[B] According to VHA's scheduling policy, when a medical appointment 
is canceled and rescheduled by the clinic, the scheduler must record 
as the desired date for the new medical appointment, the desired date 
for the original medical appointment. Three of the 19 schedulers did 
not respond to questions about rescheduling medical appointments. 

[End of table] 

During our site visits, staff at some clinics told us they change 
medical appointment desired dates to show clinic wait times within 
VHA's performance goals. A scheduler at one primary care clinic 
specifically stated that she changes the recorded desired date to the 
patient's agreed-upon appointment date in order to show shorter wait 
times for the clinic. A provider at a specialty care clinic at another 
VAMC said providers in that clinic change the desired dates of their 
follow-up appointments if a patient cannot be scheduled within the 14-
day performance goal. 

In addition, the reported wait times, derived from desired date, for 
one of the specialty care clinics we visited were inconsistent with 
the VAMC's account of appointment scheduling backlogs and scheduling 
challenges, indicating reported wait time inaccuracies. At the time of 
our site visit, officials from this clinic indicated that long waits 
for new patient appointments had existed prior to our visit and told 
us that the next available appointment for a new patient was in 6 to 8 
weeks. However, reported wait time data for the month we visited 
showed that the clinic completed all new patient appointments on the 
desired date, resulting in an unlikely high percentage of appointments 
with zero-day wait times that was inconsistent with information 
gathered during our site visit, raising questions about whether the 
desired date was recorded in accordance with VHA's scheduling policy. 
Furthermore, according to reported wait times for the VAMC, this 
clinic completed nearly all new patient appointments within 14 days of 
the desired date for the 2 months prior to our visit; and, similarly, 
in the 2 months after our visit, reported wait times for this clinic 
show completion of all new patient appointments within the 14-day time 
frame.[Footnote 33] 

VHA Officials Cited Importance of Desired Date in Capturing Patient 
Preference and Supplement Wait Time Measures with Other Information to 
Monitor Patient Access to Appointments: 

VHA central office officials told us that they recognized the 
potential reliability issues of using the desired date for measuring 
wait times, but stated that use of the desired date is the best 
approach for capturing patient experience and preference. Officials 
told us that there is no single industry standard for measuring how 
long patients wait for appointments and commonly used measures--such 
as capacity measures--do not account for patient preference or reflect 
how long the patient actually waited for an appointment.[Footnote 34] 
In addition, officials told us that the VistA scheduling system was 
not designed to capture data for management purposes, which has 
limited VHA's options for developing wait time measures. Over the 
years, VHA has tried using many different approaches to measuring wait 
times, such as capacity measures and using the date the appointment 
was created rather than the desired date to determine wait times. 
[Footnote 35] Although these measures were not officially used for 
performance accountability or reported on the PAR or NDPP and FDPP in 
fiscal year 2012, data on these measures are available to VISNs and 
VAMCs for performance monitoring. Officials told us that improving how 
wait times are measured is an ongoing effort, and they have conducted 
research to identify wait time measures that most closely correlate 
with patient satisfaction and positive outcomes. At the time of our 
review, VHA had not implemented changes to wait time performance 
measures based on the results of this research. 

In addition to measuring medical appointment wait times, VHA central 
office officials reported that VHA also uses other information to 
monitor patients' access to medical appointments and to assist VISNs 
and VAMCs in managing clinics. 

Patient Satisfaction Measures: VHA central office and VISN officials 
with whom we spoke identified patient satisfaction as another 
important indicator of patient access to medical appointments and VA 
has incorporated measures of self-reported patient satisfaction in its 
performance assessments.[Footnote 36] Specifically, the annual PAR 
includes a measure of overall patient satisfaction with VHA inpatient 
and outpatient healthcare in addition to the wait time measures 
derived from desired date. Separate measures related to patient 
satisfaction with obtaining outpatient care were also among the 
measures available for VISN and VAMC directors to include in their 
fiscal years 2011 and 2012 performance plans (NDPP and FDPP).[Footnote 
37] VHA also makes the satisfaction measures available to VISNs and 
VAMCs for continuous performance monitoring as well as available to 
the public. One of the four VAMCs we visited included the satisfaction 
measures in their performance plan for fiscal year 2012, and officials 
cited monitoring these measures on a regular basis. Officials from one 
VISN also specifically cited comparing its VAMCs' patient satisfaction 
scores to reported wait times to identify inconsistencies. However, 
the director of another VAMC said he does not rely on the satisfaction 
measures to monitor access because the data are dated by the time the 
VAMC sees the results, and instead, he relies on the scheduling data 
derived from wait time measures. 

Clinic Management Information: In addition to wait time measures, VHA 
has other information available for VISNs and VAMCs to manage clinics 
and monitor and improve clinic access, such as no-show rates and 
consult lists. Several clinic officials reported monitoring no-show 
rates--the rate at which patients do not appear for their scheduled 
appointment--in order to reduce unused appointments, for example, by 
identifying and providing additional appointment reminders to patients 
with frequent no-shows. Officials from multiple specialty clinics said 
they monitor lists of consults--requests for specialty care 
appointments--to ensure they are acted upon in a timely manner. 
[Footnote 38] Although the time between when the provider requests a 
consult and when the specialty clinic reviews the consult can affect 
the total time a patient waits for a specialty appointment, this time 
is not reflected in current wait time performance measures. 

Inconsistent Implementation of VHA's Scheduling Policy and Other 
Problems Impede VAMCs' Ability to Schedule Timely Medical Appointments: 

The four VAMCs we reviewed did not consistently implement certain 
elements of VHA's scheduling policy, including oversight requirements, 
which may result in increased wait time or delays in scheduling 
medical appointments. VAMCs also described other problems with 
scheduling timely medical appointments, including outdated technology, 
gaps in staffing of schedulers and providers, and telephone access 
problems. 

Inconsistent Implementation of VHA's Scheduling Policy Hinders VAMCs' 
Ability to Schedule Timely Medical Appointments: 

The four VAMCs we visited did not consistently implement VHA's 
scheduling policy, which is intended to facilitate the creation of 
medical appointments that meet patients' needs with no undue waits or 
delays. This policy includes the use of the VistA scheduling system to 
schedule medical appointments, and the use of the electronic wait list 
to track new patients waiting for medical appointments. (See table 3 
for information on the number of clinics we visited that did not 
implement selected elements of the VHA's scheduling policy.) 
Inconsistent implementation of VHA's scheduling policy can result in 
increased wait time or delays in obtaining medical appointments. 

Table 3: Number of Clinics That Did Not Implement Selected Elements of 
the VHA's Scheduling Policy, by VAMC Visited: 

Element of VHA's scheduling policy (number of clinic responses): Using 
VistA in scheduling of medical appointments (23); 
Number of clinics at each VAMC that did not implement the element of 
VHA's scheduling policy (clinics visited at each VAMC): 
VAMC A (4): 0; 
VAMC B (9): 1; 
VAMC C (5): 0; 
VAMC D (5): 0. 

Element of VHA's scheduling policy (number of clinic responses): 
Scheduling medical appointments while in direct communication with the 
patient (20); 
Number of clinics at each VAMC that did not implement the element of 
VHA's scheduling policy (clinics visited at each VAMC): 
VAMC A (4): 0; 
VAMC B (9): 1; 
VAMC C (5): 5; 
VAMC D (5): 0. 

Element of VHA's scheduling policy (number of clinic responses): Using 
the electronic wait list for patients who have not been seen before in 
the clinic and are waiting to be scheduled[A] (21); 
Number of clinics at each VAMC that did not implement the element of 
VHA's scheduling policy (clinics visited at each VAMC): 
VAMC A (4): 0; 
VAMC B (9): 1; 
VAMC C (5): 2; 
VAMC D (5): 1. 

Element of VHA's scheduling policy (number of clinic responses): Using 
recall/reminder software for medical appointments needed more than 3 
to 4 months into future (22); 
Number of clinics at each VAMC that did not implement the element of 
VHA's scheduling policy (clinics visited at each VAMC): 
VAMC A (4): 0; 
VAMC B (9): 0; 
VAMC C (5): 5; 
VAMC D (5): 0. 

Element of VHA's scheduling policy (number of clinic responses): 
Keeping medical appointment schedules open at least 3 to 4 months into 
future (19); 
Number of clinics at each VAMC that did not implement the element of 
VHA's scheduling policy (clinics visited at each VAMC): 
VAMC A (4): 1; 
VAMC B (9): 1; 
VAMC C (5): 1; 
VAMC D (5): 1. 

Element of VHA's scheduling policy (number of clinic responses): Total 
number instances in which elements of VHA's scheduling policy were not 
implemented; 
Number of clinics at each VAMC that did not implement the element of 
VHA's scheduling policy (clinics visited at each VAMC): 
VAMC A (4): 1; 
VAMC B (9): 4; 
VAMC C (5): 13; 
VAMC D (5): 2. 

Element of VHA's scheduling policy (number of clinic responses): Total 
number of clinics that did not implement at least one element of VHA's 
scheduling policy; 
Number of clinics at each VAMC that did not implement the element of 
VHA's scheduling policy (clinics visited at each VAMC): 
VAMC A (4): 1; 
VAMC B (9): 2; 
VAMC C (5): 5; 
VAMC D (5): 2. 

Source: GAO analysis of interviews at 23 clinics. 

Notes: Except for the element "using VistA in scheduling of medical 
appointments," we did not report responses for all 23 clinics we 
visited because interview responses were incomplete for some elements 
of the VHA's scheduling policy as depicted in the table. 

[A] Officials from 12 clinics told us that they do not use the 
electronic wait list because their clinic did not have new patients 
waiting for appointments or their clinic schedules patients for 
appointments with long wait times. 

[End of table] 

Use of VistA Scheduling System: 

One of the clinics we visited did not use the VistA scheduling system 
to determine available medical appointment dates and times, and to 
schedule medical appointments, as required by VHA's scheduling policy. 
Officials noted that this clinic lacked a full-time staff person 
dedicated to scheduling, and therefore, the providers called their 
patients to schedule their own medical appointments. Clinic staff 
reported that providers recorded medical appointments on sheets of 
paper and gave those sheets to a scheduler, who maintained a paper 
calendar of all medical appointments; this scheduler later recorded 
the appointment into the VistA scheduling system. Failing to use VistA 
to schedule medical appointments could create additional backlogs or 
scheduling errors because the schedule in VistA may not accurately 
reflect providers' availability. According to one provider in this 
clinic, for example, "staff from other departments look in VistA 
[scheduling system] and it looks like the clinic is not booked, so 
they'll send their patients as walk-in appointments. However, the 
clinic is really fully booked and patients are waiting." 

Communication with Patients: 

Officials from six clinics across two different VAMCs reported that 
staff scheduled new patient or established patient follow-up medical 
appointments without speaking to patients, and then notified patients 
of the scheduled medical appointment by letter, if the appointment was 
at least a few weeks away. This method of scheduling--referred to as 
"blind" scheduling by one official --is not in accordance with VHA's 
scheduling policy and could result in missed medical appointments for 
patients who do not receive the letters, or are not available at the 
scheduled time because patients are not involved in the scheduling 
process. One scheduler noted that he sent medical appointment letters 
because he didn't have time to call all patients to schedule 
appointments as he performs scheduling duties for 27 different 
clinics. Furthermore, outdated or incorrect patient contact 
information is an impediment to scheduling appointments via letters; 
an official in one of the six clinics told us that the databases 
containing patient contact information used to send such letters often 
do not have veterans' correct or up-to-date contact information. 

Use of the Electronic Wait List: 

Officials in four clinics across three VAMCs that had backlogs of 
patients waiting for medical appointments stated that they do not use 
the electronic wait list, the official VHA wait list used to track 
patients with whom a clinic does not have an established 
relationship.[Footnote 39] Clinics that do not use the electronic wait 
list may be at risk of losing track of new patients waiting for 
medical appointments. For example, at one specialty clinic with a 
backlog of consult requests, medical appointments for new patients 
were backed up almost 3 months; VAMC officials reported tracking 
patients waiting for medical appointments by printing paper copies of 
the consult requests from the electronic medical record. A provider at 
this clinic expressed concern that the clinic manager "has a tall 
stack of unscreened consult referrals just sitting on her desk, and no 
one is addressing them." 

Use of Recall/reminder Software: 

Officials from one VAMC stated that it did not have the required 
recall/reminder software to facilitate reminders for patients who need 
to return to the clinic for follow-up medical appointments more than 3 
to 4 months into the future; therefore, none of its clinics, including 
the five clinics that we visited, were able to use it as intended. 
[Footnote 40] Instead clinics at this VAMC use a work-around in the 
scheduling system to remind clerks to print and send letters reminding 
patients to call and schedule their follow-up medical appointments. 
However, this work-around is not automated and relies on schedulers to 
remember to generate a list of patients who need follow-up medical 
appointments, and print and send those letters. The VAMC is in the 
process of implementing recall/reminder software, according to 
officials. 

Medical Appointment Schedule Availability: 

One clinic in each of the four VAMCs visited did not keep their 
medical appointment schedules open 3 to 4 months into the future as 
required by VHA's scheduling policy.[Footnote 41] Instead, these four 
clinics allowed medical appointments to be booked only 1 to 2 months 
into the future. Limiting the future medical appointment schedule may 
limit patients' ability to schedule a follow-up medical appointment 
before leaving the clinic, as recommended by the policy, and also may 
result in additional work for clinic staff to send recall/reminder 
letters to patients for medical appointments less than 3 to 4 months 
away. 

Oversight of VHA's Scheduling Policy: 

The VAMCs we visited inconsistently implemented certain oversight 
requirements in VHA's scheduling policy--specifically, completion of 
training and certification of compliance. VAMC officials stressed the 
importance of scheduler training for ensuring correct implementation 
of VHA's scheduling policy; however, certain VAMCs did not ensure 
completion of the training by all staff who were required to complete 
it.[Footnote 42] Although all VAMCs we visited provided a list of 
staff who can schedule appointments, three VAMCs did not provide 
documentation that all staff on the list had successfully completed 
the required training. For example, officials from one VAMC stated 
that it maintained a list of staff who can schedule appointments, and 
a separate list of staff who had completed the training, but only in 
response to GAO's request for documentation did the VAMC identify 
staff with scheduling access who needed to complete the training. 
Further, three of the 19 schedulers we interviewed said they completed 
training other than the required VHA scheduler training. Completion of 
required VHA scheduler training and maintaining up-to-date 
documentation of schedulers' completion of the training is 
particularly important for ensuring consistent implementation of VHA's 
scheduling policy, given the high rates of scheduler turnover 
described by officials. 

All four of the VAMCs we visited completed the required self-
certification of compliance with the VHA's scheduling policy for 
fiscal year 2011, three of which certified overall compliance, and one 
certified overall noncompliance.[Footnote 43] However, leadership 
officials from two VAMCs, including the only one of the four that 
certified overall noncompliance, were initially uncertain who 
completed the certification or the steps taken to complete it, 
indicating that VAMCs are not always using the self-certification 
process to identify and improve problems with compliance with VHA's 
scheduling policy. 

VAMCs Identified Other Problems with Scheduling Medical Appointments, 
Including Issues with Outdated Technology, Staffing Gaps, and 
Telephone Access: 

VAMCs identified several problems that can impede the timely 
scheduling of medical appointments, which also may impact their 
compliance with VHA's scheduling policy. 

Problems with Outdated VistA Scheduling System: 

VHA central office officials and officials from all of the VAMCs we 
visited said the VistA scheduling system is outdated and inefficient, 
which hinders the timely scheduling of medical appointments. In 
particular, officials said the scheduling system requires schedulers 
to use commands requiring many keystrokes and does not allow them to 
view multiple screens at once. Schedulers must open and close multiple 
screens to check a provider's or clinic's full availability when 
scheduling a medical appointment, which is time-consuming and can lead 
to errors. For example, providers have separate schedules within VistA 
to accommodate the various types of services they provide.[Footnote 
44] Because the scheduling system cannot display multiple schedules on 
the same screen, schedulers have to enter and exit multiple screens to 
check a provider's full daily schedule when scheduling a medical 
appointment. If schedulers do not open all of the necessary screens, 
they may unknowingly create scheduling errors such as booking two 
medical appointments at the same time in different sections of a 
provider's schedule. Further, staff at one VAMC told us the problem of 
not being able to easily view a provider's full schedule can result in 
the failure to ensure that appointments are canceled when a provider 
requests it. This error could cause patients to come to the VAMC 
unnecessarily or a failure to reschedule canceled appointments in a 
timely way, both of which might lead to increased wait times for those 
patients. 

Officials from all the VAMCs we visited also noted that the VistA 
scheduling system is not easily adapted to meet clinic needs. For 
example, staff cannot create a provider schedule in the scheduling 
system that is longer than 8 hours. If a provider wants to extend his 
or her schedule on certain days, staff must create additional clinic 
schedules in the scheduling system for that provider, which can result 
in more delays and possible errors because schedulers have to check 
additional screens for medical appointment availability. Furthermore, 
officials told us that the scheduling system does not automatically 
interface with VHA's electronic medical record,[Footnote 45] which 
makes the scheduling process more time-consuming as schedulers 
alternate between the two software applications to ensure medical 
appointments are made in accordance with providers' guidance. 

VAMC officials described steps they take to ensure schedulers use 
VistA in accordance with the scheduling directive, including ongoing 
scheduler training and supervisory reviews of scheduler performance. 
However, as noted above, a lack of clarity in the desired date 
training documents and a lack of documentation of scheduler training 
at certain facilities may limit the effectiveness of these 
interventions. One VAMC provides schedulers with dual monitors to 
enable them to open multiple screens at once. Another VAMC told us 
they considered this solution in their primary care clinic, but found 
that limited physical space in the clinic did not accommodate 
additional monitors. 

In response to ongoing problems with the VistA scheduling system, VHA 
undertook an initiative to replace it in 2000, but VA abandoned the 
replacement due to weaknesses in project management and a lack of 
effective oversight. VA released a new request for information in 
December 2011 to gather information about vendors and possible 
software packages that could replace the current scheduling system. In 
September 2012, VHA told us that vendors' responses to the request for 
information indicated that VHA will be able to choose among several 
viable software packages. According to officials, VA's next step is to 
compare different vendors' software packages through the summer of 
2013, and subsequently issue a request for vendor proposals.[Footnote 
46] 

Problems with Scheduler Turnover and Provider Staffing Gaps: 

VHA central office officials and officials from all of the VAMCs we 
visited stated that shortages or turnover of schedulers also creates 
problems for the timely scheduling of medical appointments. Officials 
said that schedulers perform many important roles, including greeting 
patients, checking patients in and out of clinics, answering telephone 
calls, scheduling medical appointments for primary care, as well as 
specialty care consults, and performing other administrative support 
functions on behalf of the clinical staff. Officials explained, 
however, that high stress and a demanding workload as well as the 
entry-level pay grade of the scheduler position leads to high 
turnover. Further, officials told us that high-performing schedulers 
often are quickly promoted to other positions within VA. According to 
VHA officials, most scheduler positions are classified as a low grade 
within the government general schedule pay scale with little room for 
upward movement within the grade. Officials at two of the VAMCs we 
visited told us they are working to raise the pay level for 
schedulers; for example, one VAMC has begun to assess scheduler 
position descriptions to determine whether they can be reclassified to 
allow for more flexibility in determining scheduler salaries based on 
the variation in their assigned duties. 

Given the important role of schedulers in the scheduling process, 
officials said that even temporary staffing gaps or shortages can 
cause medical appointment delays or wait times. Staff with whom we 
spoke in several clinics said that when scheduler staffing is lacking, 
including when a scheduler is on short-term leave, it is difficult to 
cover all the scheduler's duties, and that such gaps can cause delays 
for patients. Further, we were told that scheduler staffing gaps 
resulted in inefficient use of clinical staff time. For example, at 
one specialty clinic that lacked its own scheduler, providers 
routinely scheduled their own medical appointments, which took away 
from time seeing patients, and also resulted in incorrect scheduling 
practices. Given the training needs associated with using the VistA 
scheduling system, following VHA's scheduling policy, and ensuring the 
correct use of desired date, high rates of scheduler turnover could 
contribute to inconsistent use of desired date in the scheduling 
process or other appointment scheduling problems. 

Officials at two VAMCs noted that scheduler staffing gaps are 
compounded by recent changes in their roles and responsibilities as 
VHA implements a new team-based model of primary care, which calls for 
one scheduler to be assigned to each primary care team. Officials told 
us that these changes generally increase the administrative demands 
placed on schedulers, as they are asked to respond to team duties 
while continuing to answer phones, greet patients, and register new 
patients, among other responsibilities. Officials from two VAMCs told 
us they had requested approval to hire additional staff to meet these 
added administrative needs. 

Scheduler staffing gaps may also create problems managing patient flow 
through clinics, which can impede scheduling of follow-up 
appointments, according to officials at two of the VAMCs we visited. 
Staff at these VAMCs told us that they sometimes do not have 
sufficient schedulers available to staff check-out desks, and staff at 
one VAMC added that as a result patients might "fall through the 
cracks," leaving follow-up medical appointments unscheduled unless the 
patient remembers to call in to schedule the appointment. In addition, 
when patients do not check out, schedulers are responsible for 
tracking patients needing follow-up medical appointments. This 
situation may be exacerbated in clinics that do not use the required 
recall/reminder software to facilitate the scheduling of follow-up 
medical appointments more than 3 to 4 months in the future, adding 
further to the backlog of patients in need of follow-up medical 
appointments. 

Officials from all of the VAMCs we visited told us that provider 
shortages also contribute to scheduling backlogs in certain locations 
and specialties. Recruitment and retention of providers was a 
particular challenge for VAMCs in rural areas, areas with high costs 
of living, and for certain provider specialties. All of the VAMCs we 
visited described gaps in provider staffing in certain specialty care 
clinics. Officials at all VAMCs also stated that a lack of salary 
competitiveness or the length of time to hire new providers into the 
VA system also contributed to gaps in provider staffing and scheduling 
backlogs. 

Gaps in provider staffing also can result from providers being on 
extended or unexpected leave, including vacation time, sick leave, or 
military deployments. These absences may result in longer wait times 
for patients. For example, officials at one VAMC told us that even a 
brief absence of one provider on leave can cause significant wait 
times, and that it is difficult to catch up and eliminate the backlog. 

Staff from some clinics described steps they take to reduce backlogs 
caused by gaps in provider staffing, including overbooking provider 
schedules and scheduling temporary Saturday hours. Officials at one 
VAMC told us that they employ a "floater" primary care physician to 
provide coverage for providers on leave, but an official at another 
clinic told us that they were unable to hire additional providers to 
meet the demand for medical appointments. 

Problems with Telephone Access: 

Officials at all of the VAMCs we visited told us that high call 
volumes and a lack of staff dedicated to answering the telephones 
impede the timely scheduling of medical appointments.[Footnote 47] 
Despite VHA's telephone policy requiring the provision of continuous 
telephone service for clinical care and medical appointment 
management, VAMC officials noted that schedulers are frequently 
overwhelmed by high call volumes and are unable to respond to calls in 
a timely way. In addition, officials at one VAMC told us that outdated 
telephone technology, and the lack of a dedicated VAMC-wide call 
center, limited their ability to improve their telephone 
responsiveness. VHA has reported that telephone access to VHA health 
services has historically been a frustrating experience for veterans, 
including dropped calls, multiple transfers, and long waits to reach a 
staff person able to resolve their inquiries.[Footnote 48] Further, 
patients at all of the VAMCs we visited registered complaints about 
the difficulty of reaching outpatient clinic staff by telephone and 
unreturned telephone calls. According to information on patient 
complaints provided by the four VAMCs we visited, patient complaints 
about unreturned telephone calls ranked among the top two categories 
of complaints in fiscal year 2012 at all four VAMCs.[Footnote 49] 
Further, staff at two of the VAMCs reported that their telephone calls 
to outpatient clinics within their own VAMC went unanswered, and one 
added that their inability to reach staff in their own clinics also 
was an obstacle to timely medical appointment scheduling. 

In January 2012, VHA distributed suggested best practices for 
improving telephone design, service, and access in its Telephone 
Systems Improvement Guide.[Footnote 50] This guide outlines steps VHA 
found to be effective means of improving telephone service and 
maintaining health care access, including regularly monitoring the 
purpose and volume of telephone calls; establishing dedicated staff to 
answering calls, especially at times of peak call volume; and training 
staff responsible for answering telephones in call centers. To address 
telephone issues, officials at one VAMC we visited told us they were 
developing a proposal to establish a call center with a new telephone 
system, to be staffed by schedulers dedicated to answering the 
telephones. Officials at a different VAMC stated that a scheduling 
supervisor periodically checks schedulers' telephones to ensure that 
voice mail messages are listened to and that calls are returned. 

VHA Is Implementing a Number of Initiatives to Improve Access to 
Medical Appointments: 

VHA is implementing several initiatives to improve veterans' access to 
medical appointments. Specifically, these initiatives focus on more 
patient-centered care; using technology to provide care, through means 
such as telehealth; and using care outside of VHA to reduce travel and 
wait times for veterans who are unable to receive certain types of 
outpatient care in a timely way through local VHA facilities. VHA 
officials told us they are monitoring the implementation of these 
initiatives; however, in some cases, more information is needed to 
determine their impact on timely access to care over time. 

Patient-Centered Care Initiatives: 

VHA's patient-centered medical home model for primary care, Patient 
Aligned Care Teams (PACT), is intended, in part, to improve access to 
medical appointments and care coordination through the use of 
interdisciplinary care teams and technology to communicate with 
patients. Implementation of PACT began in 2010, and is an ongoing 
effort, according to VHA officials. PACT differs from how primary care 
was previously delivered by assigning each patient to an 
interdisciplinary team. The PACT team is intended to be comprised of a 
primary care provider, registered nurse care manager, a clinical 
support staff member such as a licensed practical nurse, and a 
scheduler.[Footnote 51] These teams offer patients a centralized way 
to get questions answered by nurses or other clinical support staff 
and aim to reduce the need for face-to-face medical appointments, 
thereby enabling more efficient use of providers' time. For example, 
at one of the VAMCs we visited, patients are given a direct telephone 
number to contact their PACT team and leave a voice mail message to be 
returned by the team's registered nurse. Encouraging PACT teams' use 
of telephone communication and telephone appointments is intended to 
enable patients to more quickly obtain answers to some of their 
administrative and medical questions, such as requests for 
prescription refills, without having to schedule a face-to-face 
medical appointment. VHA officials told us that they expect PACT 
teams' use of telephone communication and telephone appointments will 
open up face-to-face medical appointment slots for patients who need 
them and might enable clinics to reduce backlogs and improve access to 
same-day primary care medical appointments. 

Officials at two VAMCs we visited told us that the transition to the 
PACT model has created some initial scheduling and staffing 
difficulties. For example, officials at these VAMCs noted that it is 
difficult for scheduling staff to respond to their PACT team duties in 
addition to meeting other responsibilities such as answering phones, 
checking in patients, registering new patients, and scheduling for 
more than one clinic. This is compounded by the fact that not all PACT 
teams have been assigned their own scheduler, as prescribed by the 
PACT model, so an individual scheduler is sometimes serving multiple 
PACT teams. Officials at these two VAMCs explained that they would 
need to hire more schedulers to meet the goal of assigning one to each 
PACT team. 

To measure the progress of PACT implementation and its impact on 
access to quality care, VHA is collecting data and tracking a series 
of measures in a monthly internal data report. Five of the PACT 
measures are (1) primary care medical appointments completed within 7 
days of the desired date;[Footnote 52] (2) same day access with 
primary care provider, or the percentage of appointments completed 
within 1 day; (3) telephone utilization, or the percentage of total 
encounters that occur by telephone; (4) continuity of care, or the 
percentage of primary care appointments with the patient's assigned 
primary care provider; and (5) post-hospital discharge contact, or 
percentage of patients discharged from the hospital who were contacted 
by their primary care provider within 2 days.[Footnote 53] As 
described earlier, accurate measurement of medical appointment wait 
times--including the first two PACT measures--is dependent upon the 
correct recording of the desired date in the VistA scheduling system. 
In fiscal year 2012, PACT measures were also included in the NDPP and 
FDPP.[Footnote 54] 

Initiatives Using Technology: 

Part of VHA's goal of achieving improved access to medical 
appointments is the increased use of technology such as telehealth and 
secure messaging.[Footnote 55] Use of these tools is intended to 
improve communication between patients and providers and open up 
providers' schedules for needed face-to-face medical appointments, 
thereby improving access to face-to-face appointments. 

VHA telehealth includes: 

* home telehealth for chronic disease management such as diabetes; 

* real-time clinic-based video telehealth, in which patients at a 
local CBOC may connect with a VHA provider at a different location to 
receive services that are unavailable at the CBOC, such as mental 
health or speech pathology; 

* and store-and-forward telehealth, in which digital images such as x-
rays or images of skin problems, are taken, stored, and sent to an 
expert for review and consultation. 

VHA officials told us that the use of telehealth can reduce both 
travel and wait times for medical appointments and help meet the needs 
of patients with chronic conditions. All VAMCs we visited told us they 
were using telehealth to improve access to care. 

Another initiative that uses technology to reduce unnecessary face-to-
face medical appointments is VHA's My HealtheVet, a web-based program 
that enables veterans to create and maintain a web-based personal 
health record with secure access to health information; services such 
as prescription refill requests; and secure messaging. Secure 
messaging allows veterans to communicate electronically with their 
health care team. According to VHA, of the more than 8 million 
veterans enrolled in VHA, 1.4 million are registered in My HealtheVet 
as of August 2012, and more than 437,000 have created secure messaging 
accounts. A recent VA study reports that secure messaging may improve 
access, patient perceptions about access, and provides for better 
communication.[Footnote 56] 

Non-VA Care Initiative: 

VHA uses non-VA care to reduce wait times and backlogs and to provide 
veterans' access to specialists not available through VHA.[Footnote 
57] Under a statutory requirement to help veterans receive care closer 
to home, VHA is piloting a new model of non-VA care known as Project 
ARCH (Access Received Closer to Home).[Footnote 58] Project ARCH is a 
five-site, 3-year pilot program administered by the VHA Office of 
Rural Health to provide health care services through contracts with 
local community providers.[Footnote 59] According to VHA officials, 
Project ARCH might help alleviate wait times for specialty care 
services with high demand, or for which there is a shortage of local 
providers. 

At the Montana Project ARCH pilot site, which we visited as part of 
our site visit to the Montana VAMC, staff from the VAMC and the 
Billings Clinic, a non-VA provider delivering services to veterans 
through Project ARCH, identified both benefits and obstacles for 
patients enrolled in Project ARCH. For example, though VAMC and 
Billings Clinic staff noted that Project ARCH reduced both travel and 
wait times for Montana veterans in need of orthopedic care, Billings 
Clinic staff also noted that difficulties in coordinating care for 
veterans moving between VHA and non-VA providers at times resulted in 
delays in providing care to those and other veterans. Additionally, 
problems with processing authorizations for certain services were 
among the concerns raised in an April 2012 evaluation of the Montana 
Project ARCH program.[Footnote 60] 

Project ARCH contractors must submit monthly reports, including 
information on medical appointment scheduling timeliness, wait times, 
and other topics. For example, the contractor for the Project ARCH 
Program in Montana is required to report on the extent to which it is 
meeting VHA's 14-day wait time goal for medical appointments--
according to VHA officials, the contractor must meet a 90 percent 
target. These wait times may not accurately reflect how long patients 
are waiting for a medical appointment, however, because the wait time 
is counted from the time the contractor receives the authorization 
from VA, rather than from the time the patient or provider requests a 
medical appointment. 

Conclusions: 

VHA officials have expressed an ongoing commitment to providing 
veterans with timely access to medical appointments and have reported 
continued improvements in achieving this goal. However, unreliable 
wait time measurement has resulted in a discrepancy between the 
positive wait time performance VA has reported and veterans' actual 
experiences. Ambiguity in what constitutes the medical appointment 
desired date--the date VHA uses as the basis for measuring wait time--
as well as manipulation of the desired date to meet goals have 
contributed to these inaccuracies. With more than 50,000 schedulers 
making approximately 80 million medical appointments in fiscal year 
2011, establishing a clear definition of the desired date or finding 
and reporting another acceptable measure of wait time is key to 
understanding how long veterans are actually waiting for medical 
appointments. Without reliable measurement of how long patients are 
waiting for medical appointments, VHA is less equipped to identify and 
address factors that contribute to wait times, or gauge the success of 
its initiatives to improve access to timely medical appointments, 
including efforts to improve primary care medical appointments. 

More consistent adherence to VHA's scheduling policy and oversight of 
the scheduling process, as well as the allocation of staffing 
resources in accordance with clinics' demands for scheduling of 
medical appointments, would potentially reduce medical appointment 
wait times. Furthermore, persistent problems with telephone access 
must be resolved to assure veterans' ability to schedule timely 
medical appointments. Ultimately, VHA's ability to ensure and 
accurately monitor access to timely medical appointments is critical 
to ensuring quality health care to veterans, who may have medical 
conditions that worsen if access is delayed. 

Recommendations for Executive Action: 

To ensure reliable measurement of veterans' wait times for medical 
appointments, we recommend that the Secretary of VA direct the Under 
Secretary for Health to take actions to improve the reliability of 
wait time measures either by clarifying the scheduling policy to 
better define the desired date, or by identifying clearer wait time 
measures that are not subject to interpretation and prone to scheduler 
error. 

To better facilitate timely medical appointment scheduling and improve 
the efficiency and oversight of the scheduling process, we recommend 
that the Secretary of VA direct the Under Secretary for Health to take 
actions to ensure that VAMCs consistently and accurately implement 
VHA's scheduling policy, including use of the electronic wait list, as 
well as ensuring that all staff with access to the VistA scheduling 
system complete the required training. 

To improve timely medical appointment scheduling, we recommend that 
the Secretary of VA direct the Under Secretary for Health to develop a 
policy that requires VAMCs to routinely assess clinics' scheduling 
needs and resources to ensure that the allocation of staffing 
resources is responsive to the demand for scheduling medical 
appointments. 

To improve timely medical appointments and to address patient and 
staff complaints about telephone access, we recommend that the 
Secretary of VA direct the Under Secretary for Health to ensure that 
all VAMCs provide oversight of telephone access and implement best 
practices outlined in its telephone systems improvement guide. 

Agency Comments and Our Evaluation: 

In reviewing a draft of this report, VA generally agreed with our 
conclusions and concurred with our recommendations. (VA's comments are 
reprinted in appendix I.) In summary, VA stated that VHA officials 
have closely followed our review and proactively taken steps in 
response to our findings. Specifically, VHA is revising and improving 
directives, policies, training, clinic management tools, and oversight 
related to scheduling practices. VA further stated that VHA is 
committed to routinely assessing clinics' scheduling needs and 
resources and developing practices and guidelines to ensure adequate 
staffing resources for scheduling medical appointments. 

VA described its plans to address each recommendation as follows: 

* In response to our recommendation that VA take actions to improve 
the reliability of wait time measures, VA concurred and stated that 
VHA will revise its scheduling policy to implement more reliable wait 
time measures and new processes to better define desired date with a 
targeted completion date of November 1, 2013. 

* In response to our recommendation that VA take actions to ensure 
that VAMCs consistently and accurately implement VHA's scheduling 
policy and ensure that all staff complete required training, VA 
concurred and stated that the revised scheduling policy will include 
improvements and standardization of the use of the electronic wait 
list. Additionally, VHA will require VISNs to update each VAMC's 
scheduler master list and verify that all schedulers on the list have 
completed required training, and will require schedulers to complete a 
standardized training update on the revised scheduling policy. The 
targeted completion date for these activities is November 1, 2013. 

* In response to our recommendation that VA develop a policy that 
requires VAMCs to routinely assess clinics' scheduling needs and 
resources, VA concurred and stated that VHA will ask VAMCs to 
routinely assess clinics' availability and ensure staff is distributed 
to meet access standards in clinics. However, VA has not specified 
requirements for VAMCs to complete these assessments nor has the 
agency provided a timeline for this process. Because schedulers are 
key to ensuring timely appointment scheduling, we believe that VA 
should establish a targeted completion date for requiring these 
assessments in policy or guidance. 

* In response to our recommendation that VA ensure that all VAMCs 
provide oversight of telephone access and implement best practices 
outlined in its telephone improvement guide, VA concurred and stated 
that VHA will require each VISN director to assess current phone 
service and develop strategic improvement telephone service plans to 
improve service. Additionally, VHA will identify a process to monitor 
performance on a quarterly basis for at least 1 year after the 
assessment. The targeted completion date for the telephone service 
assessments and plans is March 30, 2013. 

As arranged with your offices, unless you publicly announce the 
contents of this report earlier, we plan no further distribution until 
28 days after its issuance date. At that time, we will send copies of 
this report to appropriate congressional committees, the Secretary of 
Veterans Affairs and other interested parties. In addition, the report 
is available at no charge on the GAO website at [hyperlink, 
http://www.gao.gov]. 

If you or your staff have any questions about this report, please 
contact me at (202) 512-7114 or draperd@gao.gov. Contact points for 
our Offices of Congressional Relations and Public Affairs may be found 
on the last page of this report. GAO staff who made key contributions 
to this report are listed in appendix II. 

Signed by: 

Debra A. Draper: 
Director, Health Care: 

List of Requesters: 

The Honorable Jeff Miller: 
Chairman: 
Committee on Veterans' Affairs: 
House of Representatives: 

The Honorable Karen Bass: 
House of Representatives: 

The Honorable Shelley Berkley: 
House of Representatives: 

The Honorable Howard L. Berman: 
House of Representatives: 

The Honorable Brian P. Bilbray: 
House of Representatives: 

The Honorable Mary Bono Mack: 
House of Representatives: 

The Honorable Ken S. Calvert: 
House of Representatives: 

The Honorable John Campbell: 
House of Representatives: 

The Honorable Lois Capps: 
House of Representatives: 

The Honorable Judy Chu: 
House of Representatives: 

The Honorable Jim Costa: 
House of Representatives: 

The Honorable Susan Davis: 
House of Representatives: 

The Honorable David Dreier: 
House of Representatives: 

The Honorable Elton Gallegly: 
House of Representatives: 

The Honorable Joe Heck: 
House of Representatives: 

The Honorable Duncan D. Hunter: 
House of Representatives: 

The Honorable Darrell Issa: 
House of Representatives: 

The Honorable Jerry Lewis: 
House of Representatives: 

The Honorable Kevin McCarthy: 
House of Representatives: 

The Honorable Howard P. McKeon: 
House of Representatives: 

The Honorable Gary Miller: 
House of Representatives: 

The Honorable Grace Napolitano: 
House of Representatives: 

The Honorable Laura Richardson: 
House of Representatives: 

The Honorable Dana Rohrabacher: 
House of Representatives: 

The Honorable Lucille Roybal-Allard: 
House of Representatives: 

The Honorable Ed Royce: 
House of Representatives: 

The Honorable Loretta Sanchez: 
House of Representatives: 

The Honorable Adam Schiff: 
House of Representatives: 

The Honorable Brad Sherman: 
House of Representatives: 

The Honorable Henry A. Waxman: 
House of Representatives: 

[End of section] 

Appendix I: Comments from the Department of Veterans Affairs: 

Department of Veterans Affairs: 
Washington DC 20420: 

December 11, 2012: 

Ms. Debra A. Draper: 
Director, Health Care: 
U.S. Government Accountability Office: 
441 G Street, NW: 
Washington, DC 20548: 

Dear Ms. Draper: 

The Department of Veterans Affairs (VA) has reviewed the Government 
Accountability Office's (GAO) draft report, "VA Health Care: 
Reliability of Reported Outpatient Medical Appointment Wait Times and 
Scheduling Oversight Need Improvement" (GAO-13-130). VA generally 
agrees with GAO's conclusions and concurs with GAO's recommendations 
to the Department. 

The enclosure specifically addresses GAO's recommendations and 
provides general comments to the draft report. VA appreciates the 
opportunity to comment on your draft report. 

Sincerely, 

Signed by: 

John R. Gingrich: 
Chief of Staff: 

Enclosure: 

[End of letter] 

Enclosure: 

Department of Veterans Affairs (VA) Comments to Government 
Accountability Office (GAO) Draft Report "VA Health Care: Reliability 
of Reported Outpatient Medical Appointment Wait Times and Scheduling 
Oversight Need Improvement" (GA0-13-130): 

Recommendation 1: To ensure reliable measurement of veterans' wait 
times for medical appointments, we recommend that the Secretary of VA 
direct the Under Secretary of Health to take actions to improve the 
reliability of wait time measures either by clarifying the scheduling 
policy to better define the desired date, or by identifying clearer 
wait time measures that are not subject to interpretation and prone to 
scheduler error. 

VA Comment: Concur. The Veterans Health Administration (VHA) will 
revise its scheduling policy to implement more reliable wait time 
measures and new processes to better define desired date. Changes 
currently under review include: 

a. Measure "new patient wait time" as the time between the appointment 
create date and the appointment completed date. 

b. Prospectively measure "established patient wait time' from the 
patient desired date to the future scheduled appointment date. 

c. Use the patient and provider agreed upon date established in the 
exam room and documented in the Computerized Patient Record System 
electronic order to determine desired date for established patients 
returning for future visits. Pilot studies have found this process is 
not prone to interpretation or scheduler error. 

The response to Recommendation 2 describes plans for scheduler 
training to ensure policy revisions are correctly implemented. 
Targeted Completion Date: November 1, 2013. 

Recommendation 2: To better facilitate timely medical appointment 
scheduling and improve the efficiency and oversight of the scheduling 
process, we recommend that the Secretary of VA direct the Under 
Secretary of Health to take actions to ensure that VAMCs consistently 
and accurately implement VHA's scheduling policy, including use of the 
electronic wait list, as well as ensuring that all staff with access 
to the VistA scheduling system complete the required training. 

VA Comment: Concur. VHA's revised scheduling policy will include 
improvements and standardization of the use of electronic wait lists 
(EWL). 

In regard to training, VHA will require Veterans Integrated Service 
Networks (VISN) to verify and update each Facility Scheduler Master 
List. VHA will then require all schedulers to complete a standardized 
training update on the new procedures no later than (NLT) April 1, 
2013. VA's Talent Management System (TMS) will be used to verify that 
all schedulers noted on the Master List of Schedulers have completed the
mandated training. Training will include instructions about use of 
EWLs and the new scheduling requirements in the revised scheduling 
policy. Targeted Completion Date: November 1, 2013. 

Recommendation 3: To improve timely medical appointment scheduling, we 
recommend that the Secretary of VA direct the Under Secretary of 
Health to develop a policy that requires VAMCs to routinely assess 
clinics' scheduling needs and resources to ensure that the allocation 
of staffing resources is responsive to the demand for scheduling 
medical appointments. 

VA Comment: Concur. VHA will ask Medical Centers to routinely assess 
clinics' availability and ensure staff is distributed to meet access 
standards in our clinics. 

VHA will also revise and implement improved clinic management tools, 
such as an Access Index Report of key clinic operational metrics. Once 
the tools are developed, VHA will conduct training to assist clinical 
managers in making operational management decisions. 

Recommendation 4: To improve timely medical appointments and to 
address patient and staff complaints about telephone access, we 
recommend that the Secretary of VA direct the Under Secretary of 
Health to ensure that all VAMCs provide oversight of telephone access 
and implement best practices outlined in its telephone systems 
improvement guide. 

VA Comment: Concur. The DUSHOM will require each VISN Director to 
assess current phone service and develop strategic improvement 
telephone service plans (including milestones and timelines for 
implementation} to improve service. VISN and VA Medical Center (VAMC) 
leadership will use best practices, including those outlined in the 
VHA telephone systems improvement guide, to develop and implement 
these strategic improvement telephone service plans. VAMCs will also 
follow the policy related to providing telephone service for clinical 
care as outlined in VHA Directive 2007-033, Telephone Service for 
Clinical Care. The DUSHOM will identify a process to monitor 
performance on a quarterly basis for at least one year after the 
assessment. 

Targeted Completion Date: Assessments and plans including milestones 
and timelines to be completed NLT March 30, 2013, with monitoring of 
the assessments and implementation of improvements to be monitored for 
one year after the assessment. 

General Comments: 

The Veterans Health Administration (VHA) has identified that updates 
to our scheduling practices are needed. VHA officials have also 
closely followed the Government Accountability Office (GAO) review and 
proactively taken steps in response to GAO's findings. In our own 
efforts and in response to GAO, VHA is revising and improving 
directives, policies, training, clinic management tools, and oversight 
related to scheduling practices. VHA is unequivocally committed to 
providing the best care possible for Veterans and will act rapidly on 
all findings that may improve Veterans' access to health care across 
the system. 

VHA wants to emphasize that our updates to scheduling practices are 
more than a one-time effort. We are committed to routinely assessing 
clinics' scheduling needs and resources and developing practices and 
guidelines to ensure there are adequate staffing resources to be 
responsive to the demand for scheduling medical appointments. 

[End of section] 

Appendix II: GAO Contact and Staff Acknowledgments: 

GAO Contact: 

Debra A. Draper, (202) 512-7114 or draperd@gao.gov: 

Staff Acknowledgments: 

In addition to the contact named above, Bonnie Anderson, Assistant 
Director; Rebecca Abela; Jennie Apter; Rich Lipinski; Sara Rudow; and 
Ann Tynan made key contributions to this report. 

[End of section] 

Related GAO Products: 

VA Mental Health: Number of Veterans Receiving Care, Barriers Faces, 
and Efforts to Increase Access. [hyperlink, 
http://www.gao.gov/products/GAO-12-12]. Washington, D.C.: October 14, 
2011. 

Information Technology: Department of Veterans Affairs Faces Ongoing 
Management Challenges. [hyperlink, 
http://www.gao.gov/products/GAO-11-663T]. Washington, D.C.: May 11, 
2011. 

Information Technology: Management Improvements Are Essential to VA's 
Second Effort to Replace Its Outpatient Scheduling System. [hyperlink, 
http://www.gao.gov/products/GAO-10-579]. Washington, D.C.: May 27, 
2010. 

VA Health Care: Access for Chattanooga-Area Veterans Needs 
Improvement. [hyperlink, http://www.gao.gov/products/GAO-04-162]. 
Washington, D.C.: January 30, 2004. 

VA Health Care: More National Action Needed to Reduce Waiting Times, 
but Some Clinics Have Made Progress. [hyperlink, 
http://www.gao.gov/products/GAO-01-953]. Washington, D.C.: August 31, 
2001. 

Veterans' Health Care: VA Needs Better Data on Extent and Causes of 
Waiting Times. [hyperlink, 
http://www.gao.gov/products/GAO/HEHS-00-90]. Washington, D.C.: May 31, 
2000. 

[End of section] 

Footnotes: 

[1] Generally a CBOC or ambulatory care center is defined as a fixed 
health care site that is geographically distinct or separate from its 
parent VAMC. All CBOCs and ambulatory care centers generally provide 
primary care, and some may provide specialty care; services can vary 
by individual CBOC or ambulatory care center. 

[2] Outpatient clinics offer services to patients that do not require 
a hospital stay. Primary care addresses patients' routine health needs 
and specialty care is focused on a specific specialty service such as 
orthopedics or dermatology. A "clinic" may be defined as an entity for 
dividing provider workload and scheduling different types of patient 
care appointments. A particular area of care, such as primary care or 
specialty care may have multiple clinics that vary in purpose and 
size. A VAMC can provide care in each area at the VAMC or its 
affiliated CBOCs and ambulatory care centers; for example, primary 
care could be provided through multiple primary care clinics at a 
VAMC's different locations. Throughout the report we refer to a 
specific area of care at a specific location as a "clinic." 

[3] Throughout the report we will use the term "medical appointments" 
to refer to outpatient medical appointments. 

[4] GAO, VA Health Care: More National Action Needed to Reduce Waiting 
Times, but Some Clinics Have Made Progress, [hyperlink, 
http://www.gao.gov/products/GAO-01-953] (Washington, D.C.: Aug. 31, 
2001). 

[5] Department of Veterans Affairs, Office of Inspector General, Audit 
of the Veterans Health Administration's Outpatient Waiting Times, 
Report No. 07-00616-199 (Washington, D.C.: Sept. 10, 2007). 

[6] Department of Veterans Affairs, Office of Inspector General, 
Veterans Health Administration: Review of Veterans' Access to Mental 
Health Care, Report No. 12-00900-168 (Washington, D.C.: Apr. 23, 2012). 

[7] According to VHA's scheduling policy, the desired appointment 
date, referred to as the "desired date," is the date on which the 
patient or provider wants the patient to be seen. 

[8] VA assigns each VAMC a complexity score derived from multiple 
variables to measure facility complexity arrayed along four 
categories, namely patient population served, clinical services 
offered, education and research complexity, and administrative 
complexity. 

[9] We visited the highest volume CBOC for three VAMCs and the second 
highest volume for the fourth VAMC. 

[10] CBOCs and ambulatory care centers provide outpatient primary care 
and may provide specialty care services. Oversight for facility 
functions--including scheduling--occurs at the VAMC level. CBOCs and 
ambulatory care centers generally are geographically distinct from, 
but operate under the supervision of a parent VAMC, which maintains 
administrative responsibility. 

[11] During our site visits, we visited only outpatient clinics. Some 
CBOCs did not have specialty care clinics. From this point forward, we 
use VAMC to refer collectively to the VAMC and all of its affiliated 
CBOCs and ambulatory care centers. 

[12] Each of VA's 21 VISNs is responsible for managing and overseeing 
medical facilities within a defined geographic area. 

[13] VHA outpatient medical appointment scheduling policy is 
documented in VHA Directive 2010-027, VHA Outpatient Scheduling 
Processes and Procedures (June 9, 2010). We refer to the directive as 
"VHA's scheduling policy" from this point forward. 

[14] We refer to clerical or administrative support staff with 
scheduling responsibilities as "schedulers." We did not complete a 
scheduler interview for 4 of the 14 specialty clinics that we visited 
because either the clinic did not have a scheduler or it would have 
caused delays in patient care to take a scheduler off duty for an 
interview. 

[15] VHA Support Services Center maintains an internal VHA website 
that allows central office-, VISN-, and VAMC-level staff to access 
reports on a variety of topics including medical appointment data. 

[16] The electronic wait list is a type of computer software 
application designed for recording, tracking, and reporting veterans 
waiting for medical appointments. 

[17] VHA uses non-VA care to reduce wait times and backlogs and to 
provide veterans access to specialists not available through VHA. 

[18] From this point forward, the VistA medical appointment scheduling 
system will be referred to as the VistA scheduling system. 

[19] Consults--generally requests for specialty care appointments--are 
most often communicated electronically through an application in the 
electronic medical record within VistA. 

[20] VHA Directive 2007-033, Telephone Service for Clinical Care (Oct. 
11, 2007). 

[21] To obtain VHA healthcare services, veterans generally must enroll 
with VHA and register at a specific VAMC. 

[22] Specifically, VAMCs are required to maintain a list of all staff 
who can schedule medical appointments in the VistA scheduling system 
and VAMC directors are required to ensure successful completion of 
required training by all staff on the list. Schedulers are not to be 
allowed to schedule medical appointments in the VistA scheduling 
system without proof of their successful completion of this training. 

[23] For fiscal year 2011, the most recent certification available at 
the time of our review, VHA Systems Redesign collected VAMC directors' 
certification through a web-based template. As part of the 
certification, VAMC directors certify that they have completed, using 
VISN-approved processes and procedures, a yearly standardized audit of 
schedulers on the timeliness and appropriateness of scheduling actions 
and the accuracy of desired dates. 

[24] GAO, Information Technology: Management Improvements Are 
Essential to VA's Second Effort to Replace Its Outpatient Scheduling 
System, [hyperlink, http://www.gao.gov/products/GAO-10-579] 
(Washington, D.C.: May 27, 2010). 

[25] VHA also has a goal of scheduling compensation and pension 
examinations within 30 days. Compensation and pension examinations may 
be provided to veterans to establish a claim for disability 
compensation; appointment wait times for these appointments are 
outside the scope of this report. 

[26] Directors' performance contracts include measures against which 
directors are rated at the end of the fiscal year, which determine 
their performance pay. The contracts include system-wide performance 
measures, as well as individualized performance measures that are 
selected based on specific problems or needs of the respective VISN or 
VAMC. 

[27] VA prepares a congressional budget justification that provides 
details supporting the policy and funding decisions in the President's 
budget request submitted to Congress prior to the beginning of each 
fiscal year. The budget justification articulates what VA plans to 
achieve with the resources requested; it includes performance measures 
by program area. VA also publishes an annual PAR, which contains 
performance targets and results achieved against those targets in the 
previous year. 

[28] VHA officials told us the department is working to better 
coordinate consistency of the performance measures. 

[29] For example, the fiscal year 2011 NDPP and FDPP included a 
measure "percent of patients waiting for a primary care appointment 
longer than 14 days from the desired date" instead of the primary care 
measures included in fiscal year 2012.The fiscal year 2010 NDPP and 
FDPP included the measure "percent of patients waiting for a primary 
care appointment longer than 30 days from the desired date." In 
addition, the fiscal year 2011 PAR included three rather than four 
wait time measures that did not break out the new and established 
patients for primary and specialty care; the fiscal year 2012 PAR 
included separate measures for new and established patients. The 
fiscal year 2010 PAR also included three measures, one of which 
measured the "percent of new patient appointments completed within 30 
days of the appointment create date." 

[30] According to a VHA official, there were more than 50,000 staff 
across VHA who could schedule appointments at the time of our review. 

[31] Because the policy and training documents are unclear about when 
the desired date is defined by the patient or defined by the provider, 
we only identified errors related to aspects of the policy and 
training regarding how to determine and record the desired date that 
hold true despite the ambiguity. 

[32] For reporting on the PAR, VHA measures medical appointment wait 
times as the number of days between the desired date and appointment 
date. 

[33] This is based on a measure similar to the performance measure 
"percent of new patient specialty care medical appointments completed 
within 14 days of the desired date," reported on the PAR. 

[34] Clinic capacity is the supply of available future appointments. 
Capacity measures typically count the number of days between the day 
the measure is taken and the day the first or third next available 
appointment occurs. 

[35] For example, in the 2010 PAR, VA reported a wait time performance 
measure for new patient medical appointments based on days from the 
date the appointment was created. The appointment create date is 
automatically generated in the VistA scheduling system and therefore 
not prone to scheduler error. VHA officials told us that wait time 
measures based on create date do not reflect patient preference and 
therefore can incorrectly characterize wait times, particularly for 
established patient follow-up appointments which may be scheduled 
months before they are completed. 

[36] Since 2002, VHA has measured veterans' perceived access through a 
monthly Survey of Health Experiences of Patients --a survey of 
satisfaction with inpatient and outpatient care similar to the 
Department of Health and Human Services Consumer Assessment of 
Healthcare Providers Survey. 

[37] Specifically, those measures were: Getting Needed Care--combines 
responses from questions regarding how much of a problem, if any, 
patients had with various aspects of getting needed care; and Getting 
Care Quickly--combines responses from questions regarding how often 
patients received various types of care in a timely manner. 

[38] Consults are most often communicated electronically through an 
application in the electronic medical record within VistA. The 
electronic medical record is separate from the scheduling system. 

[39] According to VHA's scheduling policy, the electronic wait list is 
used to keep track of patients with whom the provider does not yet 
have an established relationship and who cannot be scheduled for 
appointments in target time frames. No other wait list formats (such 
as paper or electronic spreadsheets) are to be used for tracking 
requests for medical appointments. 

[40] Patients are entered into the recall/reminder software for the 
date they are to return to the clinic--which should be identified by 
the provider--and VistA automatically generates correspondence to the 
patient (post card or letter) a week or 2 prior to that date to remind 
the patient to call the clinic and schedule a medical appointment. 

[41] VHA's scheduling policy states that for clinics to most 
efficiently operate, "schedules must be open and available for the 
patient to make [medical] appointments at least three to four months 
into the future. Permissions may be given to schedulers to make 
appointments beyond these limits when doing so is appropriate and 
consistent with patient or provider requests. Blocking the scheduling 
of future [medical] appointments by limiting the maximum days into the 
future an appointment can be scheduled is inappropriate and is 
disallowed." 

[42] VAMCs are required to maintain a list of all staff who can 
schedule medical appointments in the VistA scheduling system, and are 
required to ensure successful completion of required VHA scheduler 
training by all staff on that list. Schedulers are not to be allowed 
to schedule medical appointments in the VistA scheduling system 
without proof of their successful completion of this training. 

[43] Of the 144 VAMCs that completed the certification for fiscal year 
2011, 109 certified overall compliance, 27 certified partial 
compliance, and 8 certified noncompliance. 

[44] For example, a physical therapist may have a separate schedule 
for amputee clinic, general physical therapy, or other types of 
services. 

[45] The electronic medical record is a component of the VistA system 
that includes patient health information and enables providers to 
record notes, such as when the provider would like to see the patient 
for a follow-up appointment, and place orders for procedures, x-rays, 
and laboratory tests, among other things. 

[46] In October 2012, VA announced a contest seeking proposals for a 
new medical appointment scheduling system from commercial software 
developers. The contest is intended to reduce risks in the future 
procurement and implementation of a new scheduling system. 

[47] VHA's policy on telephone service for clinical care, VHA 
Directive 2007-033, establishes VHA's policy of providing telephone 
access for appointment management and continuous access to health care 
advice. The telephone directive also establishes recommended 
benchmarks for telephone service at VA facilities. VAMCs differ in how 
they manage the telephones; for example, some VAMCs establish VAMC-
wide call centers to answer and direct incoming calls. VAMC telephone 
systems generally serve the VAMC including its affiliated CBOCs and 
ambulatory care centers. 

[48] Veterans Health Administration, Telephone Systems Improvement 
Guide, Second Edition (December 2011). 

[49] Each VAMC has a patient advocate who accepts and addresses 
patient complaints. The patient advocate records complaints in the 
patient advocate tracking system and tracks complaints in various 
categories, including "phone calls not returned, letters not 
answered." Two of the four VAMCs provided information on patient 
complaints for fiscal year 2012 from October 1, 2011, through May 31, 
2012. One VAMC provided information for fiscal year 2012 through June 
25, 2012, and the other provided information for fiscal year 2012 
through August 31, 2012. 

[50] Veterans Health Administration, Telephone Systems Improvement 
Guide. 

[51] VHA officials noted that even in primary care clinics in which 
PACT implementation has begun, some PACT teams are not yet fully 
staffed in accordance with the model. 

[52] In contrast to the PACT 7-day wait time measure reported in the 
NDPP and FDPP, VA reported primary care wait time performance as the 
completion of appointments within 14 days of desired date in its 
fiscal year 2012 PAR. 

[53] There are additional measures in the monthly internal data report. 

[54] VISN and VAMC directors were scored for each of the five PACT 
measures listed and rated against a sixth composite PACT measure. They 
were rated as meeting the composite PACT measure if they met targets 
for three of the five individual PACT measures. 

[55] Telehealth is the delivery of health care services using 
telecommunications technology. Using technology such as 
videoconferencing, telehealth changes the location where health care 
services are delivered. Secure messaging is VHA's web-based message 
service that allows patients to communicate nonemergency health-
related information with their health care team. 

[56] Kim Nazi, Department of Veterans Affairs Experiences with System-
wide Transformation Activities that Foster Continuous Learning and 
Improvement, Institute of Medicine Consensus Study on the Learning 
Healthcare System in America (May 2012). 

[57] Non-VA care is medical care paid for by VA but provided to 
veterans outside of VA. Non-VA care may be offered on a temporary 
basis to a veteran when medical services are not available due to a 
lack of available VA specialists, long wait times, or when VA care is 
only available at extraordinary distances from a veteran's home. VAMCs 
do not track wait times for patients using non-VA care. 

[58] See Pub. L. No. 110-387, § 403, 122 Stat. 4110, 4124 (2008). 
Veterans are eligible to participate in the program if they reside in 
a location where a pilot site is located and if they are enrolled in 
VA health care when the program starts, and meet any of the following 
criteria: live more than (1) 60 miles driving distance from the 
nearest VA health care facility providing primary care services, if 
the veteran is seeking such services; (2) 120 miles driving distance 
from the nearest VA health care facility providing acute hospital 
care, if the veteran is seeking such care; or (3) 240 miles driving 
distance from the nearest VA health care facility providing tertiary 
care, if the veteran is seeking such care. Nonenrolled veterans who 
are eligible to enroll in VA health care because they served in a 
combat theater after November 11, 1998, are also eligible to 
participate in the program. Health care delivery contracts for 
services covered under Project ARCH were awarded to Humana Veterans, a 
health services support contractor, in four pilot sites, and to a 
health care provider in the fifth pilot site. 

[59] Project ARCH services are currently being piloted at five sites, 
including Northern Maine; Farmville, Virginia; Pratt, Kansas; 
Flagstaff, Arizona; and Billings, Montana. 

[60] VA is required to evaluate the program and prepare an annual 
report to Congress for each of the 3 years of the pilot. VHA engaged a 
contractor to conduct site visits and provide VHA with quarterly 
progress reports on Project ARCH implementation. The first progress 
report for Montana VAMC's Project ARCH program was produced in April 
2012. 

[End of section] 

GAO’s Mission: 

The Government Accountability Office, the audit, evaluation, and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the 
performance and accountability of the federal government for the 
American people. GAO examines the use of public funds; evaluates 
federal programs and policies; and provides analyses, recommendations, 
and other assistance to help Congress make informed oversight, policy, 
and funding decisions. GAO’s commitment to good government is 
reflected in its core values of accountability, integrity, and 
reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO’s website [hyperlink, http://www.gao.gov]. Each 
weekday afternoon, GAO posts on its website newly released reports, 
testimony, and correspondence. To have GAO e-mail you a list of newly 
posted products, go to [hyperlink, http://www.gao.gov] and select 
“E-mail Updates.” 

Order by Phone: 

The price of each GAO publication reflects GAO’s actual cost of 
production and distribution and depends on the number of pages in the 
publication and whether the publication is printed in color or black 
and white. Pricing and ordering information is posted on GAO’s 
website, [hyperlink, http://www.gao.gov/ordering.htm]. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or 
TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card, 
MasterCard, Visa, check, or money order. Call for additional 
information. 

Connect with GAO: 

Connect with GAO on facebook, flickr, twitter, and YouTube.
Subscribe to our RSS Feeds or E mail Updates. Listen to our Podcasts.
Visit GAO on the web at [hyperlink, http://www.gao.gov]. 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 
Website: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]; 
E-mail: fraudnet@gao.gov; 
Automated answering system: (800) 424-5454 or (202) 512-7470. 

Congressional Relations: 

Katherine Siggerud, Managing Director, siggerudk@gao.gov: 
(202) 512-4400: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7125: 
Washington, DC 20548. 

Public Affairs: 
Chuck Young, Managing Director, youngc1@gao.gov: 
(202) 512-4800: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7149: 
Washington, DC 20548. 

[End of document]