This is the accessible text file for GAO report number GAO-12-214 
entitled 'Nursing Home Quality: CMS Should Improve Efforts to Monitor 
Implementation of the Quality Indicator Survey' which was released on 
March 5, 2012. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as 
part of a longer term project to improve GAO products' accessibility. 
Every attempt has been made to maintain the structural and data 
integrity of the original printed product. Accessibility features, 
such as text descriptions of tables, consecutively numbered footnotes 
placed at the end of the file, and the text of agency comment letters, 
are provided but may not exactly duplicate the presentation or format 
of the printed version. The portable document format (PDF) file is an 
exact electronic replica of the printed version. We welcome your 
feedback. Please E-mail your comments regarding the contents or 
accessibility features of this document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

[Note: This report was revised on March 9, 2012, to include omitted 
page 2 of the agency comment letter from the Department of Health and 
Human Services in appendix I, page 23] 

United States Government Accountability Office: 
GAO: 

Report to Congressional Requesters: 

February 2012: 

Nursing Home Quality: 

CMS Should Improve Efforts to Monitor Implementation of the Quality 
Indicator Survey: 

GAO-12-214: 

GAO Highlights: 

Highlights of GAO-12-214, a report to congressional requesters. 

Why GAO Did This Study: 

The Centers for Medicare & Medicaid Services (CMS) is responsible for 
ensuring that nursing homes receiving federal funding meet federal 
quality standards. CMS contracts with state survey agencies to conduct 
periodic inspections of nursing homes using a CMS-developed survey 
process. Weaknesses in the survey process led CMS to develop and begin 
implementing the Quality Indicator Survey (QIS), a new electronic 
survey. CMS developed the QIS to achieve several objectives, including 
improving the efficiency, accuracy, and consistency of the survey 
process. CMS expects the QIS to be implemented nationally by 2018. We 
examined the extent to which CMS (1) evaluates whether progress is 
being made in meeting the objectives of the QIS and (2) monitors and 
facilitates states’ implementation of the QIS. To do this, GAO 
reviewed relevant CMS documents, including CMS-commissioned studies of 
the QIS, policies, and guidance. GAO also interviewed officials from 
CMS’s Central and all 10 Regional Offices as well as officials from a 
judgmental sample of nine state survey agencies. 

What GAO Found: 

CMS has commissioned three studies to evaluate the QIS-based survey 
process but does not routinely monitor the extent to which the 
objectives established for the QIS are being met. The first two 
studies, completed in 2006 and 2007, were commissioned to determine 
whether the new survey process could be used in real-world settings 
and whether the QIS was helping the agency meet several of the 
objectives established for the process. The studies’ findings 
suggested that surveyors could use the survey process and recommended 
changes intended to help improve the QIS. CMS officials reported 
taking steps to address the studies’ findings and recommendations. In 
2009, CMS commissioned a third study that was completed in 2011 and 
identified aspects of the QIS process that could affect the 
consistency with which surveyors identify quality problems. For 
example, the study found that during resident interviews, surveyors 
did not consistently probe for further information when provided with 
incomplete responses to interview questions. However, CMS does not 
have the means to routinely monitor the extent to which the QIS is 
helping improve the survey process as intended. Such routine, ongoing 
monitoring would be consistent with federal internal control standards 
and could include the use of performance goals and measures. CMS does 
have access to some data, such as the amount of time surveyors have 
spent inspecting facilities, that could be used to help develop 
performance goals and measures. 

CMS has taken some steps to monitor and facilitate states’ 
implementation of the QIS-based routine survey, but CMS’s efforts are 
not systematic. As part of the agency’s efforts to monitor states’ 
implementation, CMS primarily uses quarterly teleconferences with 
state survey agency officials to obtain information on the extent to 
which each state has completed training all its surveyors to use the 
QIS. However, states may not always participate in the 
teleconferences, and those that do may not provide complete 
information on their progress. As a result, the information CMS 
obtains through its monitoring of states’ progress may be incomplete. 
For example, CMS was unable to provide GAO with information on 
training completion dates of all state surveyors in states that 
completed training. Although CMS reported it plans to develop a more 
systematic process for monitoring states’ implementation progress, it 
has not established a time line for doing this. To help facilitate 
states’ implementation of the QIS, CMS provides states with guidance, 
gives presentations, and offers states opportunities to share their 
implementation experiences through quarterly teleconferences. However, 
CMS does not have a systematic method for obtaining, compiling and 
sharing information on state experiences, especially information on 
approaches states have taken to help facilitate implementation of the 
QIS. Systematically sharing such information—-for example, through 
CMS’s annual conference in which all state survey agencies 
participate—-could help the agency facilitate implementation in states 
that have not begun QIS implementation. 

What GAO Recommends: 

GAO recommends that CMS develop a means to routinely monitor the 
extent to which progress is being made in meeting the objectives of 
the QIS and systematic methods for monitoring and facilitating states’ 
efforts to implement the QIS. The Department of Health and Human 
Services fully concurred with all of the recommendations. 

View [hyperlink, http://www.gao.gov/products/GAO-12-214]. For more 
information, contact Linda Kohn at (202) 512-7114 or kohnl@gao.gov. 

[End of section] 

Contents: 

Letter: 

Background: 

CMS Commissioned Studies to Evaluate the QIS, but Does Not Routinely 
Monitor Progress toward Meeting the Objectives of the QIS: 

CMS Has Taken Steps to Monitor and Facilitate States' Implementation 
of the QIS, but CMS's Efforts Are Not Systematic: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments and Our Evaluation: 

Appendix I: Comments from the Department of Health and Human Services: 

Abbreviations: 

CMS: Centers for Medicare & Medicaid Services: 

HHS: Department of Health and Human Services: 

QIS: Quality Indicator Survey: 

RTI: Research Triangle Institute International: 

SSA: Social Security Act: 

[End of section] 

United States Government Accountability Office: 
Washington, DC 20548: 

February 1, 2012: 

The Honorable Herb Kohl: 
Chairman: 
Special Committee on Aging: 
United States Senate: 

The Honorable Charles E. Grassley: 
Ranking Member: 
Committee on the Judiciary: 
United States Senate: 

Nursing homes in the United States provide skilled nursing, 
rehabilitation, and custodial care to approximately 1.5 million 
elderly and disabled individuals. Federal and state governments share 
responsibility for ensuring that the approximately 15,700 nursing 
homes in the nation receiving payment from Medicare and/or Medicaid 
programs provide quality care in a safe environment for their highly 
vulnerable residents. Specifically, the Centers for Medicare & 
Medicaid Services (CMS), an agency within the Department of Health and 
Human Services (HHS), is responsible for ensuring that nursing homes 
receiving payment from Medicare and Medicaid programs meet federal 
quality standards established pursuant to the Omnibus Budget 
Reconciliation Act of 1987.[Footnote 1] To meet this responsibility, 
CMS, among other activities, contracts with state survey agencies to 
conduct on-site nursing home inspections, which consist of routine 
surveys and complaint investigations, using a process developed by 
CMS.[Footnote 2] 

In 1998, we found weaknesses in the traditional process used to 
conduct routine surveys, including poor measurement of serious care 
problems in nursing homes.[Footnote 3] For example, we reported that 
the resident sample used in the traditional process was inadequate and 
limited a surveyors' ability to identify care problems and their 
prevalence within a facility. To address the problems we reported, and 
in response to criticism from others, CMS revised the routine survey 
process by developing the Quality Indicator Survey (QIS). According to 
CMS, the QIS was developed to help the agency meet several objectives, 
including improving surveyors' documentation of quality concerns; 
improving the efficiency of the survey process by focusing resources 
on facilities, and on areas within facilities, with the greatest 
quality concerns; and improving the accuracy and consistency with 
which surveyors identify deficiencies. 

CMS began implementation of the QIS-based routine survey process in 
2005, by launching demonstrations of the QIS in five states. To 
implement the QIS, CMS relies on the state survey agencies, as these 
agencies are largely responsible for the administration of nursing 
home inspections.[Footnote 4] To help assist state survey agencies in 
implementing the QIS, CMS provides policies, guidance and some 
training for state surveyors. CMS officials told us that as of 
September 2011, 26 states had trained or started training surveyors to 
use the QIS and 27 states had not yet started the training. CMS 
expects to complete implementation of the QIS by 2018 in all 50 
states, the District of Columbia, Puerto Rico, and the U.S. Virgin 
Islands.[Footnote 5] 

You have raised questions about CMS's efforts to implement the QIS. We 
previously provided you with information on the status of the agency's 
efforts to implement the QIS nationally.[Footnote 6] This report 
focuses on certain aspects of CMS's implementation of the QIS and will 
examine the extent to which CMS (1) evaluates whether progress is 
being made in meeting the objectives of the QIS and (2) monitors and 
facilitates states' implementation of the QIS. 

To examine the steps CMS has taken to evaluate the QIS, we reviewed 
relevant CMS documents, including the agency's plan for evaluation and 
improvement of the QIS and independent studies of the QIS commissioned 
by the agency. We also interviewed CMS officials from CMS's Central 
Office. We asked these officials for information on CMS's efforts to 
evaluate the QIS, including any efforts to routinely monitor the QIS, 
a practice consistent with federal internal control standards. 
[Footnote 7] Additionally, we interviewed the CMS contractors--Abt 
Associates Inc. and Research Triangle Institute International (RTI)--
that conducted the independent studies of the QIS. 

To examine the extent to which CMS monitors and facilitates states' 
implementation of the QIS, we reviewed CMS policies on the QIS as well 
as guidance the agency provided to state survey agencies that 
described their respective roles in implementing the QIS. We reviewed 
CMS's plan for implementing the QIS nationally. We also interviewed 
officials from CMS's Central Office and all 10 Regional Offices to 
learn more about the agency's efforts to monitor and facilitate 
states' implementation of the QIS.[Footnote 8] We also analyzed this 
information to determine whether CMS's monitoring and facilitation 
efforts are consistent with federal internal control 
standards.[Footnote 9] Additionally, we conducted structured 
interviews with state survey agency officials from a judgmental sample 
of nine states (Arizona, Colorado, Connecticut, Florida, Maryland, 
Minnesota, New York, Ohio, and West Virginia), to obtain information 
about these states' experience with implementing the QIS, including 
the extent to which CMS monitored their progress with implementation 
and helped to facilitate implementation in their state. We selected 
our sample from the 21 states that had begun implementation as of 
February 2011. These states were selected based on criteria such as 
variation in the states' QIS implementation status, the percentage of 
nursing home inspections conducted using the QIS-based process within 
a state, and state size. The findings from our structured interviews 
are limited to the nine states in our sample and are not 
representative; therefore, we will not be able to generalize these 
findings to all states. 

We conducted this performance audit from June 2011 through December 
2011 in accordance with generally accepted government auditing 
standards. Those standards require that we plan and perform the audit 
to obtain sufficient, appropriate evidence to provide a reasonable 
basis for our findings and conclusions based on our audit objectives. 
We believe that the evidence obtained provides a reasonable basis for 
our findings and conclusions based on our audit objectives. 

Background: 

Oversight of nursing homes is a shared federal-state responsibility. 
Based on statutory requirements, CMS defines standards that nursing 
homes must meet to participate in the Medicare and Medicaid programs 
and contracts with state survey agencies to assess whether homes meet 
these standards. A range of statutorily defined sanctions is available 
to CMS and the states to help ensure that homes maintain compliance 
with federal quality requirements. CMS also is responsible for 
monitoring the adequacy of state survey activities. 

Nursing Home Inspections: 

Every nursing home that receives payment from Medicare and/or Medicaid 
must undergo a nursing home inspection. These inspections consist of 
routine surveys that are conducted on average every 12 months and 
complaint investigations that are conducted in response to allegations 
of quality problems.[Footnote 10] During a routine survey, teams of 
state surveyors--generally consisting of registered nurses, social 
workers, dieticians, or other specialists--evaluate a nursing home's 
compliance with federal nursing home quality standards. There are 
approximately 200 different standards that focus on the delivery of 
care, resident outcomes and facility conditions. The standards are 
grouped into 15 categories, such as Quality of Life, Resident 
Assessment, Quality of Care, and Administration. For example, there 
are 23 standards within the Quality of Care category ranging from the 
prevention of pressure sore development to keeping the resident 
environment as free of accident hazards as is possible. During nursing 
home inspections, state surveyors may cite deficiencies--areas in 
which facilities fail to meet federal quality standards--and nursing 
homes must prepare a correction plan to address most 
deficiencies.[Footnote 11] State surveyors may also conduct what are 
known as revisit surveys to ensure that the homes corrected or took 
steps to correct identified deficiencies. 

To ensure the reliability of state survey agencies' inspections of 
nursing homes, federal survey teams composed of CMS staff from its 10 
Regional offices conduct what are known as federal monitoring surveys, 
which include comparative and observational surveys. In a comparative 
survey, a federal survey team conducts an independent inspection of a 
home recently surveyed by a state survey agency in order to compare 
and contrast its findings with those of the state survey team. In an 
observational survey, federal surveyors accompany a state survey team 
to a nursing home to evaluate the team's on-site survey performance 
and ability to document deficiencies. The federal monitoring surveys 
are to be conducted annually in at least 5 percent of state-surveyed 
Medicare and Medicaid nursing homes in each state.[Footnote 12] CMS 
has developed the QIS-based tool for routine surveys and plans to 
develop QIS-based tools for all other types of nursing home 
inspections: revisit surveys, complaint investigations and federal 
monitoring surveys.[Footnote 13] 

QIS Process: 

The QIS process is similar to the traditional survey method.[Footnote 
14] For example, both the QIS and the traditional survey process 
involve analyzing information available about the nursing home 
facilities and their residents before entering the nursing home in 
order to identify areas of care that have been historically 
problematic and residents who may be at higher risk for poor care; 
interviewing residents, family members, and staff; observing staff and 
nursing home practices; reviewing medical records; and citing 
deficiencies when necessary. However, the QIS differs from the 
traditional survey in two key areas: 

* The QIS provides an electronic means of conducting nursing home 
inspections. The collection, recording and analysis of information and 
documentation of findings are done electronically using a tablet 
computer; the traditional method is paper-based. 

* The QIS software draws an expanded random sample of up to 70 
residents for surveyors to interview, observe, and review their 
medical records during the inspection. In contrast, the traditional 
survey method allows surveyors to review various reports, data sets, 
and guidance and use their judgment to select a sample of residents, 
consisting of about 20 percent of the current resident census. 

CMS Commissioned Studies to Evaluate the QIS, but Does Not Routinely 
Monitor Progress toward Meeting the Objectives of the QIS: 

CMS has commissioned three studies to evaluate the QIS, an electronic 
tool that CMS developed to assess the quality of nursing home care, 
and has taken some steps to improve the QIS in response to the 
findings of the studies. However, the agency does not routinely 
monitor the extent to which progress is being made in meeting the 
objectives established for the QIS.[Footnote 15] 

CMS Has Commissioned Studies to Evaluate Various Aspects of the QIS: 

The first CMS-commissioned study was completed by Abt Associates Inc. 
in 2006 and tested the QIS under real-world conditions. This study 
analyzed survey data available from three states in which CMS had 
begun to conduct demonstrations of the QIS: Connecticut, Kansas, and 
Ohio. The study was intended to examine surveyors' experiences using 
the QIS to conduct actual nursing home inspections, and identify ways 
to improve the QIS. In order to meet this general purpose, the study 
assessed several areas, including whether surveyors were able to use 
an electronic tool to conduct routine surveys in a real-world setting, 
whether surveyors could complete the routine survey using the QIS in 
the same amount of time that surveyors needed to complete routine 
surveys using the traditional methodology, and whether any changes 
were needed to improve the QIS tool. Among other results, the study 
found that state surveyors were able to use the QIS-based process to 
conduct routine surveys and generally perceived the process to be an 
improvement over the traditional survey method. However, the study 
also found that surveyors generally needed more time to complete 
routine surveys using the QIS tool rather than the traditional survey 
method.[Footnote 16] The study cited several potential reasons for 
this time difference, ranging from problems with the QIS software and 
hardware to forms within the QIS survey process that could be made 
less time-consuming. These findings led the researchers to recommend 
ways to streamline and otherwise improve the QIS tool in order to 
reduce the amount of time surveyors needed to complete the QIS-based 
routine survey. CMS revised the QIS survey tool and process based on 
many of these recommendations. These revisions included providing more 
guidance on certain aspects of the QIS process during QIS training; 
eliminating certain duplicative requirements for collecting 
information; and altering various survey questions, for example, by 
adjusting the wording of survey questions to make them more 
understandable to nursing home residents and reducing the time 
required for surveyors to explain them. 

In 2007, Abt Associates Inc. completed the second CMS-commissioned 
study that evaluated the use of the QIS-based routine survey process 
in the field. This study included survey data from all five 
demonstration states (California, Connecticut, Kansas, Louisiana, and 
Ohio) and was intended to evaluate, among other things, whether the 
QIS achieved certain of the agency's objectives for the process. 
[Footnote 17] These objectives included providing a routine survey 
tool that improves surveyor accuracy--that is, helps surveyors better 
identify deficiencies; improves surveyor efficiency--that is, helps 
surveyors focus their time on facilities with the greatest number of 
quality concerns; and improves surveyors' documentation of 
deficiencies. In terms of whether the QIS led to improved accuracy or 
improved documentation of deficiencies, the report found that the QIS-
based routine survey and the traditional survey approach generally led 
to comparable results. The researchers, therefore, concluded that the 
QIS did not significantly enhance or diminish surveyors' ability to 
accurately identify or document deficiencies. Similarly, the study 
suggested that in general, surveyors using the QIS tool completed 
routine surveys as efficiently, but not more efficiently, as surveyors 
using the traditional survey methodology.[Footnote 18] The study made 
several recommendations to CMS for improving the QIS process, 
including that CMS further evaluate how well the QIS protocols use 
preliminary information gathered during the survey to identify problem 
areas in the facility for more in-depth investigation. Further, the 
study recommended that CMS clarify the agency's investigative 
protocols, which CMS publishes and makes available within the QIS 
software to help surveyors assess nursing home compliance with federal 
quality standards.[Footnote 19] Following completion of this study, 
CMS decided to implement the QIS nationally and CMS officials told us 
they worked to improve aspects of the QIS process by taking steps to 
address the study's findings and recommendations.[Footnote 20] 

In 2009, CMS commissioned a third study to evaluate the QIS-based 
routine survey process and make recommendations for improvement. This 
study, conducted by RTI and completed in 2011, was intended to 
identify aspects of the QIS process that could affect how consistently 
surveyors identify quality problems and to suggest ways to improve 
these aspects of the process. This study included survey data from 19 
states that had begun to implement the QIS.[Footnote 21] The study 
found that various aspects of the QIS process, including the way 
resident interviews were conducted and the surveyor initiation process 
(in which surveyors identify additional problem areas, not identified 
by the QIS software, for further investigation during surveys), might 
affect the consistency with which surveyors identify quality problems 
during nursing home inspections.[Footnote 22] For example, the study 
found that surveyors did not consistently probe for further 
information when residents gave incomplete answers to interview 
questions, and that some surveyors were likely to initiate problem 
areas during surveys on their own, while others relied mainly on the 
QIS software to identify areas for investigation. The study also 
identified potential methods for addressing these issues and improving 
the consistency of the QIS process, including providing additional 
guidance to surveyors on interviewing techniques and developing 
additional standardized training for new surveyors to ensure they are 
familiar with federal regulations. CMS officials told us that they 
intend to consider the recommendations made in this study in their 
efforts to continue improving the QIS. 

CMS Has Not Established the Means to Routinely Monitor Progress toward 
Meeting the Objectives of the QIS, Such as Developing Performance 
Goals and Measures: 

Although CMS has commissioned past studies to evaluate the QIS, the 
agency does not monitor the QIS on an ongoing basis. Such ongoing 
monitoring of the QIS could allow CMS to collect data to routinely 
evaluate the extent to which progress is being made in meeting all of 
the objectives established for the QIS to improve the survey process 
as intended. Monitoring, a key component of federal internal control 
standards, helps provide management with reasonable assurance that an 
agency or its programs and initiatives are achieving effectiveness and 
efficiency of operations and compliance with applicable laws and 
regulations, among other factors.[Footnote 23] One way agencies can 
monitor program operations on an ongoing basis to ensure that the 
programs are meeting their objectives is through the use of 
performance goals and measures. Performance measurement involves 
identifying performance goals and measures, establishing performance 
baselines by tracking performance over time, identifying targets for 
improving performance, and measuring progress against those targets. 
[Footnote 24] In the case of the QIS-based routine survey process, 
information collected through performance goals and measures could 
help CMS routinely monitor the extent to which the objectives of the 
QIS are being achieved. This information could also inform future 
efforts by CMS to modify and improve the QIS process as needed. CMS 
officials acknowledged the need for certain performance measures--
specifically with regard to examining the effect of the QIS on 
surveyor consistency--but noted that they did not have performance 
goals or measures in place as of December 2011. This lack of 
performance goals and measures can affect CMS's ability to effectively 
monitor the QIS. 

CMS currently collects some data that would allow it to begin 
monitoring aspects of the performance of the QIS. For example, the 
agency currently collects information from routine surveys--such as 
data that describes the amount of time surveyors have spent inspecting 
facilities, as well as the deficiencies that surveyors cited during 
each inspection--that could be used to help measure the efficiency of 
the QIS process.[Footnote 25] CMS could use these data to conduct 
analyses of surveyor efficiency, similar to the analysis conducted by 
Abt Associates Inc., and derive efficiency scores that could be used 
for monitoring.[Footnote 26] These efficiency scores could also be 
used to measure changes in surveyor efficiency over time, with the 
goal of attaining an agency-established target efficiency score. 

Additionally, CMS could use data from desk audit reports to help 
measure progress towards the objective of improving surveyor 
consistency. According to CMS officials, desk audit reports are 
standardized reports that are generated and sent at established 
intervals to Regional Offices and state survey agencies.[Footnote 27] 
These reports contain data related to surveys, including data intended 
to describe surveyor performance during each QIS-based routine survey 
conducted. Specifically, desk audit reports contain data on 
approximately 30 different aspects of QIS surveys, such as the problem 
areas that were identified for further investigation during a survey 
and the number of residents for whom a surveyor did not document 
complete information during a survey. The reports also indicate how 
these data compare to rates at the state, regional, or national level. 
[Footnote 28] CMS officials told us that these reports are intended to 
serve as a tool to help officials from Regional Offices and state 
survey agencies identify and respond to inconsistencies in QIS 
surveyor performance. Since data in the reports can illustrate 
variation in how surveyors use the QIS, they could also be used as a 
starting point for CMS to create performance goals and measures for 
consistency and help the agency routinely evaluate whether progress is 
being made toward the objective of improved consistency. 

CMS Has Taken Steps to Monitor and Facilitate States' Implementation 
of the QIS, but CMS's Efforts Are Not Systematic: 

CMS has taken some steps to monitor states' progress implementing the 
QIS, but the agency's monitoring of states' efforts is not systematic. 
As a result, CMS has incomplete information on states' implementation 
progress. CMS has also taken steps to facilitate states' 
implementation of the QIS, such as holding quarterly teleconferences 
for all states implementing the QIS, but the agency lacks a systematic 
method for obtaining, compiling, and sharing information on states' 
experiences implementing the QIS. 

CMS Has Taken Some Steps to Monitor States' Progress Implementing the 
QIS but Does Not Have a Systematic Method of Monitoring: 

CMS has taken some steps to monitor states' progress implementing the 
QIS-based routine survey, but these efforts are not systematic. As 
part of its national implementation plan, CMS established three 
milestones to guide states' implementation of the QIS-based routine 
survey.[Footnote 29] These milestones include (1) start dates by which 
each state is to begin training state surveyors to use the QIS, (2) 
completion of training for all surveyors within three years of the 
training start date, and (3) use of the QIS for all routine surveys by 
state surveyors after all surveyors in the state have completed 
training. CMS policy requires that all routine surveys in a state 
should be conducted using the QIS after all surveyors in a state have 
been trained. CMS officials told us that the agency uses various 
methods to periodically monitor states' progress on the three 
milestones. 

* Monitoring of state training start dates--CMS officials said that 
they track start dates for the initial training of state surveyors to 
use the QIS-based routine survey; however, the information that CMS 
obtains only pertains to a small group of surveyors in each state. 
Specifically, CMS arranges and pays for a contractor to train a small 
group of surveyors in states to (1) conduct routine surveys using the 
QIS-based process and (2) become registered QIS trainers who will be 
responsible for training the rest of surveyors in the state.[Footnote 
30] CMS officials told us the training contractors are responsible for 
providing CMS with information on the start dates of states' training 
of this small group of surveyors. 

* Monitoring of state surveyor training completion dates--CMS 
officials told us that they periodically monitor the extent to which 
each state completes training of all its surveyors to use the QIS-
based process, but CMS's primary method of obtaining this information 
is not systematic and may not allow CMS to collect complete 
information. CMS officials were unable to provide us with information 
on the training completion dates of all state surveyors in those 
states that the agency considered to have completed training. To 
monitor training completion, CMS officials told us that they review 
information on the number of state surveyors trained to use the QIS 
and compare the time between states' training start dates and when 
states inform CMS that they have trained all of their surveyors. CMS 
officials told us they obtain this training completion information 
primarily through the agency's quarterly teleconference with state 
survey agency officials implementing the QIS, a forum where states 
provide information on their implementation progress, including status 
of state surveyor training completion. However, we found that the 
information CMS obtains through this method may be incomplete because 
(1) states may not participate in these teleconferences and (2) some 
states that participate may not provide complete information on their 
implementation progress. For example, we observed a quarterly 
teleconference in July 2011, during which at least 7 out of the 24 
states that participated in the call did not provide the agency with 
an update on state surveyor training completion during the 
teleconference. 

* Monitoring the use of the QIS--CMS officials said that they review 
routine survey data to track the number of routine surveys performed 
using the QIS-based process; however, CMS may not have complete 
information to help determine the extent to which states are using the 
QIS. Specifically, agency officials said they periodically review 
whether state surveyors are conducting routine surveys using the QIS-
based process exclusively after states complete training of all state 
surveyors to use the QIS.[Footnote 31] To do this, CMS officials pull 
information on the number of surveys completed using the QIS tool from 
their nursing home survey database. However, because CMS may not have 
complete information on the training completion dates for all state 
surveyors, agency officials may not be able to accurately determine 
the date after which states should have begun conducting routine 
surveys exclusively using the QIS-based process.[Footnote 32] 

CMS acknowledges that the agency does not have a systematic process in 
place that would allow the agency to obtain comprehensive information 
on states' progress implementing the QIS. The ability to perform 
effective, ongoing monitoring in the course of normal operations is an 
integral element of managing a program, according to federal internal 
control standards.[Footnote 33] CMS officials told us that while they 
continue to periodically monitor states' progress on the QIS 
milestones using various means, the agency plans to develop a more 
systematic process for monitoring states' progress toward completion 
of state surveyor training and use of the QIS-based process for 
routine surveys. According to CMS, this method will provide a way for 
the agency to centrally collect information directly from the states, 
including information on states' completion of surveyor training. The 
agency plans to use this information to help monitor states' overall 
progress toward implementing the QIS nationally. However, officials 
said the agency has not yet established a time line for the 
development of this method. 

CMS Has Taken Some Steps to Facilitate States' Implementation of the 
QIS, but Efforts to Obtain and Share Information on States' 
Experiences Are Not Systematic: 

CMS has taken steps to facilitate states' implementation of the QIS. 
The agency has issued periodic guidance to state survey agencies and 
conducted presentations on the QIS process at various venues, 
including CMS's annual conference with state survey agency officials. 
In addition, as stipulated by its guidance, CMS provides opportunities 
for state survey agencies to discuss implementation issues and 
concerns with the agency and other states. CMS achieves this in part 
through its quarterly teleconferences for all states currently 
implementing the QIS. During these teleconferences, state survey 
agency officials have opportunities to share their experiences 
implementing the QIS, including how they address any challenges they 
encounter. CMS also provides updates and clarification of agency 
policies that are to guide state survey agencies as they implement the 
QIS during the teleconferences. For example, during the July 2011 CMS 
quarterly teleconference with state survey agencies, agency officials 
updated state survey agency officials on potential changes that the 
agency was considering regarding its policy on the number of surveyors 
assigned to conduct routine surveys using the QIS process.[Footnote 
34] Further, the CMS Central Office and CMS Regional Office staff 
provides information and assistance on an as-needed basis to 
individual states. Through these efforts, CMS officials stated they 
learn of common challenges states face during implementation, such as 
problems with using QIS software. The agency has provided some 
assistance to states to resolve these challenges, including providing 
guidance and implementing QIS software upgrades. 

Despite the agency's efforts, CMS does not have a systematic method 
for obtaining, compiling, and sharing information with state survey 
agencies on states' experiences with implementing the QIS. According 
to federal internal control standards, an agency should establish 
internal controls that help ensure adequate communication and 
information sharing with key external stakeholders that may 
significantly affect the agency's ability to meet its goals.[Footnote 
35] CMS obtains and shares information with states primarily through 
the quarterly teleconferences or direct assistance provided upon 
request to state survey agencies--the agency's key external 
stakeholders. While states share some information on their 
implementation experiences during CMS's quarterly teleconferences, 
such disclosures are voluntary. Additionally, participation in the 
teleconferences is limited to only states that are implementing the 
QIS. Further, states may not always participate in these 
teleconferences, so information communicated during these 
teleconferences may not reach all states. While CMS officials 
summarize the teleconference proceedings in the form of meeting 
minutes and share these minutes with meeting participants, the agency 
does not share this information with state survey agencies that have 
not yet begun implementation of the QIS. 

We learned that some states have independently developed various 
approaches to help their state facilitate implementation of the QIS. 
While some states have shared these approaches with CMS, the agency is 
not systematically analyzing, compiling, and sharing these efforts 
with all other states. 

* Officials in some states helped facilitate implementation by taking 
the initiative to complete state surveyor QIS training in less than 3 
years. For example, some state survey agency officials told us they 
worked to efficiently train their surveyors to use the QIS-based 
process so that their survey workload would not be affected.[Footnote 
36] 

* One state survey agency facilitated state surveyor training 
regionally by training surveyors in areas of the state with high 
concentrations of large nursing home facilities first. Officials said 
that they took this approach because it allowed them to efficiently 
balance their survey workload. 

* All nine state survey agencies we spoke to reported technology-
related issues that state surveyors encountered when using the QIS 
software and tablet computers during nursing home inspections. 
[Footnote 37] State survey agency officials from two states reported 
inviting state survey agency information technology staff to 
participate in QIS surveyor training in order to enhance the staff's 
ability to resolve the technological issues surveyors expected to 
encounter when using the QIS software and tablet computers. 

* Some state survey agency officials we spoke to reported that some 
surveyors in their state experienced difficulties with using QIS 
software and tablet computers due to a learning curve or a lack of 
computer skills needed to operate these tools. For example, officials 
from four states reported developing additional training materials 
related to basic computer skills or general information on the QIS in 
order to improve state surveyors' proficiency with using these QIS 
tools. 

Providing a systematic method--such as using CMS's annual conference 
in which all state survey agencies participate--to share information 
on state approaches to facilitate implementation and resolve 
implementation challenges could help the agency facilitate 
implementation in states that have begun implementation or will 
implement the QIS in the future. 

Conclusions: 

CMS developed the QIS in order to improve nursing home inspections, 
which are used to help ensure the quality of nursing home care 
delivered each year to a vulnerable population of more than one 
million people. After conducting demonstrations and evaluating the 
results of studies, CMS determined that the QIS should be implemented 
nationwide by 2018, using a phased implementation. To date, CMS has 
made progress implementing the QIS in 26 states. As this 
implementation continues over the next 6 years, it is important that 
CMS know the extent to which the implementation of the QIS is 
improving the survey process and whether states are making progress 
toward and meeting CMS's implementation milestones. 

Although CMS commissioned several studies of the QIS and has shown a 
willingness to adapt the QIS process based on some study findings, the 
agency has not established the means--such as performance goals and 
measures--that would allow for routine, ongoing monitoring of the 
extent to which the QIS is helping CMS improve the survey process as 
intended. Such performance goals and measures could be developed in 
part using data CMS currently collects. Routine monitoring that 
includes performance goals and measures for the QIS could also help 
CMS continue to identify areas for improvement as CMS officials expect 
to continue to make changes intended to improve the QIS throughout its 
phased implementation. While studies commissioned at key points can be 
valuable, the information CMS would obtain on the QIS through routine 
monitoring is needed for this ongoing effort. 

CMS also does not have reasonable assurance that the phased 
implementation across the states is making sufficient progress in 
order to meet the 2018 goal. Although CMS uses various means to 
monitor and facilitate states' efforts to implement the QIS, these 
efforts are not systematic. As a result, CMS lacks complete 
information on the extent to which states have met their 
implementation milestones, including the extent to which states are 
using the QIS tool once all surveyors in the state have completed 
training. CMS recognizes that it needs a more systematic methodology 
for monitoring states' progress, but has not yet implemented it. To 
help facilitate implementation of the QIS, CMS provides the 
opportunity for all states implementing the QIS to discuss their 
implementation challenges; however, CMS could better leverage the 
experience of the states by systematically obtaining, compiling, and 
sharing strategies states have used to address their challenges 
implementing the QIS, for example sharing such information through 
CMS's annual conference in which all state survey agencies 
participate. Such information could help the remaining states that 
have not yet begun implementation to learn from the experience of 
others. 

Recommendations for Executive Action: 

As CMS continues to implement the QIS, we recommend the Administrator 
of CMS take the following three actions: 

* develop a means--such as performance goals and measures--to 
routinely monitor the extent to which CMS is making progress in 
meeting the objectives established for the QIS; 

* develop and implement a systematic methodology to track state survey 
agencies' progress with implementation activities; and: 

* develop and implement a systematic method for obtaining, compiling, 
and sharing information from state survey agencies about their 
implementation experiences. 

Agency Comments and Our Evaluation: 

On behalf of CMS, HHS reviewed a draft of this report and provided 
written comments, which appear in appendix I. In its written comments, 
HHS fully concurred with all three of our recommendations and said 
that it would work to implement them expeditiously. Specifically, HHS 
said that CMS intends to: 

* Establish new or modify existing measures or processes to more 
effectively monitor CMS's progress towards meeting QIS goals and 
objectives. HHS informed us that CMS has begun to review available 
data related to the QIS--for example, data on survey workload, survey 
deficiencies, and the number of surveys performed--to more effectively 
monitor the QIS and guide improvement efforts. 

* Formalize the data collection method used to track states' progress 
with QIS implementation activities. According to HHS, the agency plans 
to institute an automated process to obtain updated information on 
states' progress with training surveyors which, when combined with 
existing data, could provide a more accurate assessment of 
implementation activities in any state. 

* Enhance existing and add new methods of sharing information on the 
QIS with states. HHS stated that CMS plans to expand existing 
information sharing opportunities--for example, quarterly calls and 
presentations at annual meetings--to be available to all states, not 
just those in the process of implementing the QIS. It will also give 
consideration to using web-based capability to facilitate sharing 
information on QIS implementation. 

In addition to the comments related to our recommendations, HHS noted 
that CMS has reassessed the timetable for QIS expansion to additional 
states due to budgetary concerns and the QIS will not be introduced to 
additional states in fiscal year 2012 as planned. According to HHS, 
CMS will instead focus efforts on more effectively implementing the 
QIS in those states where training of state surveyors has begun or has 
been completed, addressing barriers to QIS implementation, and 
completing the remaining components of the QIS. This reassessment of 
the QIS timeline raises concerns that nationwide implementation of the 
QIS will extend beyond 2018, as currently planned. The longer it takes 
the agency to implement the QIS nationwide, the longer it will take to 
fully realize the intended benefits of the QIS--improving the nursing 
home inspection process and helping to ensure the quality of nursing 
home care delivered each year to more than one million people. 

As agreed with your offices, unless you publicly announce the contents 
of this report earlier, we plan no further distribution until 30 days 
from the report date. At that time, we will send copies of this report 
to the Secretary of HHS, the Administrator of CMS, and relevant 
congressional committees. In addition, the report will be available at 
no charge on the GAO website at [hyperlink, http://www.gao.gov]. 

If you or your staffs have any questions regarding this report, please 
contact me at (202) 512-7114 or kohnl@gao.gov. Contact points for our 
Offices of Congressional Relations and Public Affairs may be found on 
the last page of this report. Key contributors to this report were E. 
Anne Laffoon, Assistant Director; George Bogart; Krister Friday; Seta 
Hovagimian; Samantha Poppe; and Priyanka Sethi. 

Signed by: 

Linda T. Kohn: 
Director, Health Care: 

[End of section] 

Appendix I: Comments from the Department of Health and Human Services: 

Department of Health & Human Services: 
Office of The Secretary: 
Assistant Secretary for Legislation: 
Washington, DC 20201: 

January 13, 2012: 

Linda T. Kohn: 
Director, Health Care: 
U.S. Government Accountability Office: 
441 G Street NW: 
Washington, DC 20548: 

Dear Ms. Kohn: 

Attached are comments on the U.S. Government Accountability Office's 
(GAO) draft report entitled, "Nursing Home Quality: CMS Should Improve 
Efforts to Monitor Implementation of the Quality Indicator Survey" 
(GAO 12-214). 

The Department appreciates the opportunity to review this report prior 
to publication. 

Sincerely, 

Signed by: 

Jim R. Esquea: 
Assistant Secretary for Legislation: 

Attachment: 

[End of letter] 

General Comments Of The Department Of Health And Human Services (HHS) 
On The Government Accountability Office's (GAO) Draft Report Entitled, 
"Nursing Home Quality: CMS Should Improve Efforts To Monitor 
Implementation Of The Quality Indicator Survey" (GAO-12-214): 

The Department appreciates the opportunity to review and comment on 
this draft report. 

The GAO recommendations focus primarily on performance measurement and 
monitoring. This is an important aspect of the full set of coordinated 
endeavors necessary for successful Q1S development and deployment. 
Those endeavors include the following: 

1. Design and Development of the System; 
2. Resource and Investment Strategy; 
3. On-going Management; 
4. Training and Deployment; 
5, Performance Measurement, Monitoring and Improvement. 

The Centers for Medicare & Medicaid Services (CMS) continue to find 
the QIS to represent a significant advance in many dimensions of the 
survey process, particularly in terms of its potential to increase 
consistency and objectivity. Recent CMS design work has also created 
QIS capability to provide management information that can continuously 
promote survey process improvement on the part of both State managers 
and CMS. We therefore, continue to move forward on all aspects of the 
QIS enterprise. 

At the same time, the significant difference between the final fiscal 
year (FY) 2012 appropriation for Medicare survey & certification 
functions and the President's budget request has caused us to reassess 
the timetable for QIS expansion to additional States. In light of both 
the current budget limitations and future uncertainties, we recently 
communicated to States that we will extend the timeline for adding new 
States to the QIS. We do not expect that any additional States will 
begin QIS anew in FY 2012, beyond those States that have already 
initiated the training process. We will instead concentrate on fully 
and effectively implementing the QIS in those States that are already 
implementing the system, overcoming barriers to implementation on the 
part of QIS State or Federal oversight survey teams, and completing 
the remaining design features of the system. 

The CMS fully concurs with all the GAO recommendations, and will work 
to implement them expeditiously. 

GAO Recommendation No. 1: 

Develop a means-—such as performance goals and measures-—to routinely 
monitor the extent to which CMS is making progress in meeting the 
objectives established for the QIS. 

CMS Response: 

CMS concurs. We believe that by establishing new or modifying existing 
measures or processes, we can more effectively and systematically 
monitor progress toward program goals and objectives. 

As part of our internal QIS program review, CMS has begun the process 
of reviewing specific data attributes, including several identified in 
the draft report. Moreover, we believe that the following specific 
attributes present excellent opportunities for new measures that we 
can use more effectively to monitor progress: 

1) Survey workload data (CMS Form 670). 

2) Survey deficiency data (number, type, and makeup of deficiencies). 

3) Total number and type of surveys performed. 

4) Desk audit report use. 

We will continue to review survey deficiency data on a quarterly basis 
to evaluate differences between the traditional and QIS processes. 
Analysis of these data will reveal improvement opportunities that will 
guide us to a more consistent survey process. 

Additional measures will permit a broad view of implementation status, 
such as State progress toward implementation within a three year time 
period. The percentage of surveys that are of the type QIS compared to 
traditional surveys, provides one such indication of implementation 
progress. When we look at survey data in combination with surveyor 
data (as outlined in GAO recommendation 2), CMS is able to review the 
implementation criteria for a given State and determine if there are 
any inconsistencies in the data (such as the number of surveyors 
trained versus the type and number of surveys performed). We plan to 
link survey and surveyor data as a metric to analyze whether State 
agencies (SAs) are allowing non-QIS trained surveyors to perform QIS 
surveys, as well as to determine whether QIS-registered surveyors are 
performing traditional surveys. 

As outlined in the report, all of these data attributes can play a key 
role in helping CMS determine whether or not we are meeting our 
programmatic goals and objectives as well as allowing continuous 
improvement in the QIS process. 

GAO Recommendation No. 2: 

Develop and implement a systematic methodology to track State survey 
agencies' progress with implementation activities. 

CMS Response: 

CMS concurs. We propose to formalize the data collection related to 
tracking QIS surveyors and certified trainers within SAs to accomplish 
this recommendation. Currently, CMS defines "fully implemented" as 
"100 percent of the State's surveyors have been trained in the QIS and 
the SA is only performing surveys of type QIS once that status has 
been achieved."

Based upon GAO's recommendation, CMS plans to institute a formal 
process based on data in our Learning Management System (LMS) that 
contains an electronic record of each surveyor's training. CMS will 
require QIS-participating SAs to provide and update information for 
all surveyors in the State as to their QIS training status, as well as 
to provide CMS with the dates that surveyors completed the 
requirements qualifying them to become QIS-registered surveyors and/or 
a QIS-certified trainers. 

To date, CMS has been relying on a paper or verbal process for the SAs 
to provide updated information on the status of their training, 
including the number of surveyors trained in the QIS during our QIS 
all State quarterly call. The new process will allow CMS to calculate 
an overall SA implementation completion percentage based upon LMS 
electronic data, and to generate and review reports at regular 
intervals. 

Formalizing and automating the training data will allow CMS to 
calculate the percentage of surveyors in all QIS States that are QIS-
registered and qualified. In addition, we can quickly determine how 
many trainers each State currently has ready. CMS also can compare 
these data to a measure identifying the percentage of QIS surveys 
performed in a State so as to establish a complete picture of the 
implementation status in any State. Finally, CMS can use these data to 
determine if SAs are meeting initial implementation time frames, as 
well as to provide an ongoing measure of program stability. 

GAO Recommendation No. 3: 

Develop and implement a systematic method for obtaining, compiling and 
sharing information from State survey agencies about their 
implementation experiences. 

CMS Response: 

CMS concurs. We recognize the need to be able to more effectively 
obtain, publish, and disseminate all information relevant to SA QIS 
implementation experiences. Currently, CMS uses several methods to 
provide information to QIS States. CMS will enhance those processes as 
well as add the following new capabilities to accomplish this 
recommendation: 

1) Expand QIS all State quarterly calls to include all interested SAs 
so they can begin to learn about the QIS implementation process and 
other SA experiences as early as possible. 

2) Continue to hold QIS meetings and presentations at the annual 
Survey & Certification Training and Leadership Summit. We will, 
however, emphasize that all SAs would benefit from participation in 
these meeting or sessions rather than limiting attendance to those SAs 
that already have begun the implementation process. We will share all 
new information on the overall QIS program at these meetings, 
including processes that may have been achieved as part of this 
reporting process. 

3) Continue to hold QIS meetings and presentations at the Association 
of Health Facility Survey Agencies annual conferences. We will 
emphasize that all presentations and information presented would be 
beneficial to all survey agencies, not necessarily just those that 
have begun the QIS implementation process. Moreover, we will look 
forward to the opportunity to share all new information relative to 
the overall QIS program at these meetings including processes that may 
have been achieved as part of this reporting process. 

4) Propose to create a new or modify an existing web-based capability 
to enable and simplify the process of sharing and publishing all 
information relative to QIS implementation. CMS envisions the 
functionality will provide the SAs with more sophisticated 
capabilities for document sharing, notification, and publication. CMS 
envisions the functionality to be similar to what users have 
experienced using other web sites of this nature, for example a 'Share 
Point' portal. 

CMS believes that incorporating all of these changes will afford all 
SAs, not just those implementing the QIS, the opportunity to more 
effectively research, document, and understand the challenges and 
lessons learned about QTS implementation. 

[End of section] 

Footnotes: 

[1] Pub. L. No. 100-203, Title IV, Subtitle C, §§ 4201 et seq., 101 
Stat. 1330, 1330-160, et seq. (amending Titles XVIII and XIX of the 
Social Security Act). According to CMS, combined Medicare and Medicaid 
payments for nursing home care in 2010 were about $91 billion. 

[2] Under contract with CMS, state survey agencies conduct inspections 
of health care facilities--such as nursing homes, home health 
agencies, intermediate care facilities for the mentally retarded, and 
hospitals--that participate in Medicare and Medicaid. 

[3] See GAO, California Nursing Homes: Care Problems Persist Despite 
Federal and State Oversight, [hyperlink, 
http://www.gao.gov/products/GAO/HEHS-98-202] (Washington, D.C.: July 
27, 1998). 

[4] State survey agencies administer and have discretion over many 
survey activities and policies, including hiring and retaining a 
surveyor workforce, training surveyors, and conducting supervisory 
reviews of surveys. For example, state survey agencies are largely 
responsible for training all surveyors in their states to use the QIS; 
in 2011, there were roughly 4,700 surveyors nationwide. 

[5] In this report, we use the term "states" to refer to this group of 
states and territories. 

[6] See GAO, Nursing Home Quality: Implementation of the Quality 
Indicator Survey, [hyperlink, http://www.gao.gov/products/GAO-11-403R] 
(Washington, D.C.: Apr. 6, 2011). 

[7] See GAO, Standards for Internal Control in the Federal Government, 
[hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1] 
(Washington, D.C.: November 1999). 

[8] CMS's Regional Offices are located in Atlanta, Boston, Chicago, 
Dallas, Denver, Kansas City, New York, Philadelphia, Seattle, and San 
Francisco. 

[9] See [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1]. 

[10] Every nursing home is required to receive an inspection not less 
than once every 15 months and the statewide average interval for these 
surveys must not exceed 12 months. 42 C.F.R. § 488.308 (2010). 

[11] For serious deficiencies, CMS generally imposes sanctions, which 
may result in the termination of a nursing home's participation in the 
Medicare and Medicaid programs. 

[12] Under the Social Security Act (SSA), HHS is required to conduct a 
certain number of federal validation surveys, also referred to as 
federal monitoring surveys, each year. In no case is HHS to survey 
less than a total of five nursing homes in a state per year. SSA 
§1819(g)(3)(B)(Medicare) (codified at 42 U.S.C. §1395i-3(g)(3)(B)); 
SSA § 1919(g)(3)(B)(Medicaid) (codified at 42 U.S.C. § 1396r(g)(3)(B)). 

[13] Revisit surveys ensure that nursing homes remediate certain 
deficiencies. A complaint investigation generally focuses on a 
specific allegation regarding a resident's care or safety. Federal 
monitoring surveys are done to ensure the reliability of state survey 
agencies' routine surveys. 

[14] See [hyperlink, http://www.gao.gov/products/GAO-11-403R] for more 
details on similarities and differences between the QIS and the 
traditional survey processes. 

[15] According to CMS, the objectives of the QIS include improving 
surveyors' documentation of deficiencies; improving the efficiency of 
the routine survey process by focusing survey resources on facilities 
and areas within facilities where the greatest quality concerns exist; 
and improving the accuracy and consistency with which surveyors 
identify deficiencies. 

[16] According to the report, the researchers were unable to draw firm 
conclusions regarding this comparison due to data limitations such as 
the limited number of QIS surveys conducted at the time. 

[17] Although California participated in the five-state study, CMS 
allowed the state to postpone further implementation of QIS due to 
budgetary concerns within the state. 

[18] The researchers noted that the study's findings were limited by 
factors such as small sample size and an inability to control for 
certain external factors, such as surveyor skills, that may have 
affected results of the study. 

[19] These protocols, called critical element pathways, assist 
surveyors in completing a systematic review of problematic areas in 
facilities that are selected for in-depth investigation during the 
survey. For example, the critical element pathway for the federal 
standard regarding the prevention and treatment of pressure ulcers in 
nursing facilities provides a checklist of items for the surveyor to 
assess as well as questions, such as whether the facility 
comprehensively assessed the resident's condition, for the surveyor to 
answer during this portion of the survey. 

[20] According to CMS officials, the agency considered the results of 
the second study, among other factors, when making the decision to 
continue implementation of the QIS on a national scale. 

[21] The study included both qualitative and quantitative analyses. 
Some analyses in the study used data from fewer than 19 states. 

[22] According to CMS, the QIS software uses preliminary survey data 
to select problematic areas in facilities for further investigation 
during surveys. However, surveyors have the option to initiate the 
selection of areas for investigation outside of those selected by the 
QIS software. Surveyors may rely on their professional experience and 
judgment to make these selections. 

[23] The five components of internal control are Control Environment, 
Risk Assessment, Control Activities, Information and Communications, 
and Monitoring. See also GAO, Standards for Internal Control in the 
Federal Government, [hyperlink, 
http://www.gao.gov/products/GAO/AIMD-00-21.3.1] (Washington, D.C.: 
November 1999). 

[24] See Pub. L. No. 103-62, § 4, 107 Stat. 285, 287 (generally 
codified, as amended, to 31 U.S.C. §1115-1117). 

[25] More specifically, CMS data that could be used to measure 
efficiency include the number, scope, and severity of deficiencies 
cited during QIS-based surveys, as well as the resources, in terms of 
the amount of time or number of staff, required to complete these 
surveys. 

[26] Abt Associates Inc. indicated that a lack of QIS data limited 
their ability to draw firm conclusions about the effect of the QIS on 
surveyor efficiency. This study was conducted when CMS first began to 
implement the QIS and included data from five states. As of September 
2011, CMS reported that 26 states have begun QIS implementation. 
Therefore, more QIS data should be currently available for analysis. 

[27] The reports are generated by a CMS contractor and sent to 
Regional Offices on a quarterly basis and to state survey agencies 
generally after every 10 QIS-based routine surveys conducted in the 
state. 

[28] CMS officials noted that a version of the report for the CMS 
Central Office is under development; however, the agency has not yet 
established a time line for completing this work. 

[29] CMS has established a phased approach for implementing the QIS 
nationally. The agency has grouped states into six cohorts. States 
within a cohort are to meet the milestones for beginning and 
completing surveyor training and using the QIS for all routine surveys 
within approximately the same general timeframe. According to CMS 
officials, the agency has rolled out the QIS in three of the six 
cohorts as of September 2011. 

[30] To begin training of state surveyors, states select an initial 
core group of eight surveyors--consisting of two survey teams of four 
surveyors each--to be trained by the CMS contractor to use the QIS. It 
is up to state survey agencies to train all remaining surveyors within 
CMS's 3 year implementation time frame. 

[31] Under CMS policy, once all of a state's surveyors are trained in 
QIS that state is expected to conduct all routine surveys using QIS. 
CMS officials acknowledged that not all states reporting to have 
completed training of all surveyors may be using the QIS exclusively. 

[32] State survey agencies may continue to use the traditional 
methodology to conduct nursing home inspections until all surveyors in 
that state have been trained to use the QIS. Additionally, those 
states that have not yet begun QIS implementation continue to use the 
traditional methodology. 

[33] See [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1]. 

[34] According to CMS, agency policy previously required a minimum of 
four state surveyors per team to conduct routine surveys using the QIS 
process, regardless of the size of the nursing home facility. As of 
August 2011, CMS revised this policy to allow states to determine the 
number of surveyors per team for facilities with fewer than 40 beds. 

[35] See [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1]. 

[36] Some state survey agency officials told us that QIS training can 
affect state survey workloads. Because surveyors' completion of QIS 
training required them to be offline for at least 4 weeks, fewer state 
surveyors were then available to conduct surveys for this period 
during the implementation process. 

[37] These technological issues included software malfunctions that 
caused surveyors' tablet computers to crash or erase survey data 
collected during nursing home inspections. 

[End of section] 

GAO’s Mission: 

The Government Accountability Office, the audit, evaluation, and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the 
performance and accountability of the federal government for the 
American people. GAO examines the use of public funds; evaluates 
federal programs and policies; and provides analyses, recommendations, 
and other assistance to help Congress make informed oversight, policy, 
and funding decisions. GAO’s commitment to good government is 
reflected in its core values of accountability, integrity, and 
reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO’s website [hyperlink, http://www.gao.gov]. Each 
weekday afternoon, GAO posts on its website newly released reports, 
testimony, and correspondence. To have GAO e mail you a list of newly 
posted products, go to [hyperlink, http://www.gao.gov] and select “E-
mail Updates.” 

Order by Phone: 

The price of each GAO publication reflects GAO’s actual cost of 
production and distribution and depends on the number of pages in the 
publication and whether the publication is printed in color or black 
and white. Pricing and ordering information is posted on GAO’s 
website, [hyperlink, http://www.gao.gov/ordering.htm]. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or 
TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card, 
MasterCard, Visa, check, or money order. Call for additional 
information. 

Connect with GAO: 

Connect with GAO on facebook, flickr, twitter, and YouTube.
Subscribe to our RSS Feeds or E mail Updates. Listen to our Podcasts.
Visit GAO on the web at [hyperlink, http://www.gao.gov]. 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 
Website: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]; 
E-mail: fraudnet@gao.gov; 
Automated answering system: (800) 424-5454 or (202) 512-7470. 

Congressional Relations: 

Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-4400
U.S. Government Accountability Office, 441 G Street NW, Room 7125
Washington, DC 20548. 

Public Affairs: 
Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800
U.S. Government Accountability Office, 441 G Street NW, Room 7149 
Washington, DC 20548.