This is the accessible text file for GAO report number GAO-06-54 
entitled 'Hospital Quality Data: CMS Needs More Rigorous Methods to 
Ensure Reliability of Publicly Released Data' which was released on 
February 24, 2006. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as part 
of a longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

Report to the Committee on Finance, U.S. Senate: 

United States Government Accountability Office: 

GAO: 

January 2006: 

Hospital Quality Data: 

CMS Needs More Rigorous Methods to Ensure Reliability of Publicly 
Released Data: 

GAO-06-54: 

GAO Highlights: 

Highlights of GAO-06-54, a report to the Committee on Finance, U.S. 
Senate: 

Why GAO Did This Study: 

The Medicare Modernization Act of 2003 directed that hospitals lose 
0.4 percent of their Medicare payment update if they do not submit 
clinical data for both Medicare and non-Medicare patients needed to 
calculate hospital performance on 10 quality measures. The Centers for 
Medicare & Medicaid Services (CMS) instituted the Annual Payment Update 
(APU) program to collect these data from hospitals and report their 
rates on the measures on its Hospital Compare Web site. 

For hospital quality data to be useful to patients and other users, 
they need to be reliable, that is, accurate and complete. GAO was asked 
to (1) describe the processes CMS uses to ensure the accuracy and 
completeness of data submitted for the APU program, (2) analyze the 
results of CMS’s audit of the accuracy of data from the program’s first 
two calendar quarters, and (3) describe processes used by seven other 
organizations that assess the accuracy and completeness of clinical 
performance data. 

What GAO Found: 

CMS has contracted with an independent medical auditing firm to assess 
the accuracy of the APU program data submitted by hospitals, but has no 
ongoing process in place to assess the completeness of those data. 
CMS’s independent audit checks accuracy by comparing the quality data 
submitted by hospitals from the medical records for a sample of five 
patients per calendar quarter for each hospital to the quality data 
that the contractor has reabstracted from the same records. The data 
are deemed to be accurate if there is 80 percent or greater agreement 
between these two sets of results. CMS has established no ongoing 
process to check data completeness. For the payment updates for fiscal 
years 2005 and 2006, CMS compared the number of cases submitted by a 
hospital to the number of Medicare claims that hospital submitted. 
However, these analyses did not address non-Medicare patient records, 
and the approach that CMS took in these analyses was not capable of 
detecting incomplete data for all hospitals. 

Although GAO found a high overall baseline level of accuracy when it 
examined CMS’s assessment of the data submitted for the first two 
quarters of the APU program, the results are statistically uncertain 
for up to one-third of hospitals, and a baseline level of data 
completeness cannot be determined. The median accuracy score of 90 to 
94 percent—depending on the calendar quarter and measures used—was well 
above the 80 percent accuracy threshold set by CMS, and about 90 
percent of hospitals met or exceeded that threshold for both the first 
and the second calendar quarters of 2004. However, for approximately 
one-fourth to one-third of all the hospitals that CMS assessed for 
accuracy, the statistical margin of error for their accuracy score 
included both passing and failing accuracy levels. Consequently, for 
these hospitals, the small number of cases that CMS examined was not 
sufficient to establish with statistical certainty whether they met the 
accuracy threshold set by CMS. With respect to completeness of data, 
CMS did not assess the extent to which all hospitals submitted data on 
all eligible patients, or a representative sample thereof, for the two 
baseline quarters. As a result, there were no data from which to derive 
an assessment of the baseline level of completeness of the quality data 
that hospitals submitted for the APU program. 

Other reporting systems that collect clinical performance data have 
adopted a range of activities to ensure data accuracy and completeness, 
which include some methods employed by all, such as checking the data 
electronically to identify missing data. Officials from some of the 
other reporting systems and an expert in the field stressed the 
importance of including an independent audit in the methods used by 
organizations to check data accuracy and completeness. Most of the 
other reporting systems incorporate three methods into their process 
that CMS does not use in its independent audit. Specifically, most 
include an on-site visit in their independent audit, focus their audits 
on a selected number of facilities, and review a minimum of 50 patient 
medical records during the audit. 

What GAO Recommends: 

GAO recommends that CMS take steps to improve its processes for 
ensuring the accuracy and completeness of hospital quality data. In 
commenting on a draft of this report, CMS agreed to implement steps to 
improve the quality and completeness of the data. 

www.gao.gov/cgi-bin/getrpt?GAO-06-54. 

To view the full product, including the scope and methodology, click on 
the link above. For more information, contact Cynthia A. Bascetta, 
(202) 512-7101 or BascettaC@gao.gov. 

[End of section] 

Contents: 

Letter: 

Results in Brief: 

Background: 

CMS Has Processes for Checking Data Accuracy but Has No Ongoing Process 
to Check Completeness: 

Data Accuracy Baseline Was High Overall, but Statistically Uncertain 
for Many Hospitals, and Data Completeness Baseline Cannot Be 
Determined: 

Other Reporting Systems Use Various Methods to Ensure Data Accuracy and 
Completeness, Notably an Independent Audit: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments: 

Appendix I: Scope and Methodology: 

Appendix II: Other Reporting Systems: 

Appendix III: Data Tables on Hospital Accuracy Scores: 

Appendix IV: Comments from the Centers for Medicare & Medicaid 
Services: 

Appendix V: GAO Contact and Staff Acknowledgments: 

Tables: 

Table 1: HQA Hospital Quality Measures: 

Table 2: Percentage and Number of Hospitals Whose Baseline Accuracy 
Score Met or Fell Below the 80 Percent Threshold, by Measure Set and 
Quarter: 

Table 3: Background Information on CMS and Other Reporting Systems: 

Table 4: Processes Used by CMS and Other Reporting Systems to Ensure 
Data Accuracy: 

Table 5: Processes Used by CMS and Other Reporting Systems to Ensure 
Data Completeness: 

Table 6: Median Hospital Baseline Accuracy Scores, by Hospital 
Characteristic, Quarter, and Measure Set: 

Table 7: Proportion of Hospitals with Baseline Accuracy Scores Not 
Meeting 80 Percent Threshold, by Hospital Characteristic, Quarter, and 
Measure Set: 

Table 8: Percentage of Hospitals with Baseline Accuracy Scores Not 
Meeting 80 Percent Threshold, by JCAHO-Certified Vendor Grouped by 
Number of Hospitals Served, Quarter, and Measure Set: 

Table 9: Breadth of Confidence Intervals in Percentage Points Around 
the Hospital Baseline Accuracy Scores at Selected Percentiles, by 
Measure Set and Quarter: 

Table 10: For Hospitals with Confidence Intervals That Included the 80 
Percent Threshold, Percentage of Total Hospitals with an Actual 
Baseline Accuracy Score That Either Met or Failed to Meet the 
Threshold, by Measure Set and Quarter: 

Figures: 

Figure 1: Approximate Times for Collection, Submission, and Reporting 
of Hospital Quality Data: 

Figure 2: Baseline Hospital Accuracy Scores at Selected Percentiles, by 
Measure Set and Quarter: 

Figure 3: Percentage of Hospitals Whose Baseline Accuracy Score 
Confidence Intervals Clearly Exceed, Fall Below, or Include the 80 
Percent Threshold, by Measure Set and Quarter: 

Abbreviations: 

ACC: American College of Cardiology: 

AMI: acute myocardial infarction: 

APU program: Annual Payment Update program: 

CABG: coronary artery bypass grafting: 

CAP: community-acquired pneumonia: 

CDAC: Clinical Data Abstraction Center: 

CMS: Centers for Medicare & Medicaid Services: 

DAVE: Data Assessment and Verification Project: 

HF: heart failure: 

HQA: Hospital Quality Alliance: 

IFMC: Iowa Foundation for Medical Care: 

JCAHO: Joint Commission on Accreditation of Healthcare Organizations: 

MDS: Minimum Data Set: 

MEDPAR: Medicare Provider Analysis and Review: 

MMA: Medicare Prescription Drug, Improvement, and Modernization Act: 

MSA: metropolitan statistical area: 

NCQA: National Committee for Quality Assurance: 

PCI: percutaneous coronary intervention: 

PTCA: percutaneous transluminal coronary angioplasty: 

QIO: quality improvement organization: 

SPARCS: Statewide Planning and Research Cooperative System: 

SSA: Social Security Administration: 

STS: Society of Thoracic Surgeons: 

United States Government Accountability Office: 

Washington, DC 20548: 

January 31, 2006: 

The Honorable Charles E. Grassley: 
Chairman: 
The Honorable Max Baucus: 
Ranking Minority Member: 
Committee on Finance: 
United States Senate: 

The Medicare Prescription Drug, Improvement, and Modernization Act 
(MMA) of 2003 created a financial incentive for hospitals to submit 
data to provide information about their quality of care that could be 
publicly reported.[Footnote 1] Under Section 501(b) of MMA, acute care 
hospitals shall submit the clinical data from the medical records of 
all Medicare and non-Medicare patients needed to calculate hospitals' 
performance on 10 quality measures. If a hospital chooses not to submit 
the data, it will lose 0.4 percent of its annual payment update from 
Medicare for a subsequent fiscal year.[Footnote 2] The Centers for 
Medicare & Medicaid Services (CMS) established the Annual Payment 
Update program (APU program)[Footnote 3] to implement this provision of 
MMA. Participating hospitals submit quality data that are used to 
calculate a hospital's performance on the measures quarterly,[Footnote 
4] according to a schedule defined by CMS. MMA affects hospital annual 
payment updates for fiscal year 2005 through fiscal year 2007.[Footnote 
5] For fiscal year 2005, the first year of the program, CMS based its 
annual payment update on quality data submitted by hospitals for 
patients discharged between January 1, 2004, and March 31, 2004. 

Under MMA, the 10 quality measures for which hospitals report data are 
those established by the Secretary of Health and Human Services as of 
November 1, 2003. The measures cover three conditions: heart attack, 
heart failure, and pneumonia. Over 3 million patients were admitted to 
acute care hospitals in 2002 with these three conditions, representing 
approximately 10 percent of total acute care hospital admissions. For 
patients over 65, acute care hospital admissions for the three 
conditions represented approximately 16 percent of total admissions. 

The collection of quality data on the 10 measures is part of a larger 
initiative to provide useful and valid information about hospital 
quality to the public.[Footnote 6] In April 2005, CMS launched a Web 
site called "Hospital Compare" to convey information on these and other 
hospital quality measures to consumers. Additional measures are being 
introduced by CMS,[Footnote 7] and it is expected that public reporting 
of hospital quality measures will continue into the future. Hospitals 
may submit quality data on additional measures for the APU program, but 
CMS bases any reduction in the annual payment update on the 10 measures 
referenced in the MMA. In addition to this effort, other public and 
private organizations also administer reporting systems in which 
clinical data are collected and may be released to the public. 

In order for publicly released information on the hospital quality 
measures to be useful to patients, payers, health professionals, health 
care organizations, regulators, and other users, the quality data used 
to calculate a hospital's performance on the measures need to be 
reliable, that is, both accurate and complete. If a hospital submits 
complete data, that is, data on all the cases that meet the specific 
inclusion criteria for eligible patients, but the data are not 
collected, or abstracted, from the patients' medical records 
accurately, the data will not be reliable. Similarly, if a hospital 
submits accurate data, but those data are incomplete because the 
hospital leaves out eligible cases, the data will not be reliable. Data 
that are not reliable may present a risk to people making decisions 
based on the data, such as a patient choosing a hospital for treatment. 
The program's initial, or baseline, data could describe data 
reliability at the start of the program and provide a reference point 
for any subsequent assessments. 

You asked us to provide information on the reliability of publicly 
reported information on hospital quality obtained through the APU 
program. In this report, we (1) describe the processes CMS uses to 
ensure that the quality data submitted by hospitals for the APU program 
are accurate and complete and any plans by CMS to modify its processes; 
(2) determine the baseline levels of accuracy and completeness for the 
data for patients discharged from January 2004 through June 2004, the 
first two quarters of data submitted by hospitals under the APU 
program; and (3) describe the processes used by seven other 
organizations that collect clinical performance data to assess the 
accuracy and completeness of quality data for selected reporting 
systems. 

In addressing these objectives, we collected information through 
interviews, examination of documents, and data analysis. To describe 
CMS's processes for ensuring the accuracy and completeness of the 
quality data for the APU program, we interviewed program officials from 
CMS and its contractors,[Footnote 8] hospital associations, quality 
improvement organizations (QIO), and hospital data vendors.[Footnote 9] 
In addition, we examined both publicly available and internal documents 
from CMS and its contractors. To determine the baseline accuracy and 
completeness of data submitted for the APU program, we drew on 
available information collected by CMS. In particular, we analyzed the 
accuracy of the quality data based on the reabstraction of patient 
medical records performed by CMS's Clinical Data Abstraction Center 
(CDAC).[Footnote 10] The reabstraction results available at the time we 
conducted our analyses pertained to hospital discharges that took place 
from January 1, 2004, through June 30, 2004.[Footnote 11] We extracted 
additional information about hospitals from the Medicare Provider of 
Services database, including the number of Medicare-certified beds and 
urban or rural location. After examining the CDAC data and reviewing 
the procedures that CMS has put in place to conduct the reabstraction 
process, we determined that the data were sufficiently reliable to use 
in estimating the baseline level of accuracy characterizing the quality 
data submitted by hospitals for those two calendar quarters. Regarding 
data on completeness of the quality data, we interviewed CMS officials 
and contractors and examined related documents. To examine the methods 
used by other reporting systems[Footnote 12] to assess data 
completeness and accuracy, we conducted structured interviews with 
officials from seven organizations,[Footnote 13] including government 
agencies, that administer such systems. We focused on reporting systems 
that collect clinical rather than administrative data. We selected a 
mix of systems, in terms of public or private sponsorship, types of 
providers assessed, and medical conditions covered, to ensure variety. 
We also spoke with individual health professionals with expert 
knowledge in the field of hospital quality assessment. 

Our analysis of the level of accuracy and completeness of the quality 
data is based on the procedures developed by CMS to validate the data 
submitted; we have not independently compared the data submitted by 
hospitals to the original patient clinical records. In addition, we did 
not assess the performance of hospitals with respect to the quality 
measures themselves (which show how often the hospitals provided a 
specified service or treatment when appropriate). We conducted our work 
from November 2004 through January 2006 in accordance with generally 
accepted government auditing standards. For more details on our scope 
and methodology, see appendix I. 

Results in Brief: 

CMS has processes for ensuring the accuracy of the quality data 
submitted by hospitals for the APU program, but has no ongoing process 
for assessing the completeness of those data. To check accuracy, one 
CMS contractor electronically checks the data as they are submitted to 
the clinical warehouse, and another operates CMS's CDAC that conducts 
an independent audit by sampling five patient record abstractions from 
all the quality data submitted by each hospital in a quarter. CDAC then 
compares the quality data originally collected by the hospital from the 
medical records for those five patients to the quality data it has 
reabstracted from the same medical records. The data are deemed to be 
accurate if there is 80 percent or greater agreement between these two 
sets of results. CMS did not require hospitals to meet the 80 percent 
threshold for the 10 APU measures to receive their full annual payment 
update for fiscal year 2005. However, for fiscal year 2006, CMS reduced 
the payment update by 0.4 percentage points for hospitals whose data on 
the APU measures do not meet the 80 percent threshold. To assess 
completeness, CMS has twice compared the number of cases submitted by 
each hospital for the APU program for a given period to the number of 
claims each hospital submitted to Medicare, once for the fiscal year 
2005 update and once for the fiscal year 2006 update. However, these 
analyses did not address non-Medicare patient records, and the approach 
that CMS took in these analyses was not capable of detecting incomplete 
data for all hospitals. For example, to determine which hospitals could 
receive the full fiscal year 2006 update, CMS limited its analysis to 
hospitals that submitted no patient data at all to the clinical 
warehouse in a given quarter. CMS has not put in place an ongoing 
process for checking the completeness of the data that hospitals submit 
for the APU program that would provide accurate and consistent 
information for all patients and all hospitals. Nor has CMS required 
hospitals to certify that they submitted data for all eligible patients 
or a representative sample thereof. 

We could determine a baseline level of accuracy for the quality data 
submitted by hospitals for the APU program but not a baseline level of 
completeness. We found a high overall baseline level of accuracy when 
we examined CMS's assessment of the data from the first two calendar 
quarters of 2004. Overall, the median accuracy score exceeded 90 
percent, which was well above the 80 percent accuracy threshold set by 
CMS, and about 90 percent of hospitals met or exceeded that threshold 
for both the first and the second calendar quarters of 2004. For most 
hospitals whose accuracy score was well above the threshold, the 
results based on the reabstraction of five cases were statistically 
certain. However, for approximately one-fourth to one-third of all the 
hospitals that CMS assessed for accuracy, the statistical margin of 
error for their accuracy score included both passing and failing 
accuracy levels. Consequently, for these hospitals, the five cases that 
CMS examined were not sufficient to establish with statistical 
certainty whether the hospital met the threshold level of data 
accuracy. Accuracy did not vary between rural and urban hospitals, and 
small hospitals provided data as accurate as those from larger 
hospitals. The completeness baseline could not be determined because 
CMS did not assess the extent to which all hospitals submitted data on 
all eligible patients, or a representative sample thereof, for the 
first two calendar quarters of 2004, and consequently there were no 
data from which to derive such an assessment. 

Other reporting systems that collect clinical performance data have 
adopted various methods to ensure data accuracy and completeness. Some 
of these methods are used by all of these other reporting systems, such 
as checking the data electronically to identify missing data. Officials 
from some of the other systems and an expert in the field stressed the 
importance of including an independent audit in the methods used by 
organizations to check data accuracy and completeness. Most other 
reporting systems that conduct independent audits incorporate three 
methods as part of their process that CMS does not use in its 
independent audit. Specifically, most include an on-site visit, focus 
their audits on a selected number of facilities or reporting entities, 
and review a minimum of 50 patient medical records per reporting entity 
during the audit. 

In order for CMS to ensure that the hospital quality data are accurate 
and complete, we recommend that the CMS Administrator, focusing on the 
subset of hospitals for which it is statistically uncertain if they met 
CMS's accuracy threshold in one or more previous quarters, increase the 
number of patient records reabstracted by CDAC. We further recommend 
that CMS require hospitals to certify that they took steps to ensure 
that they submitted data on all eligible patients, or a representative 
sample thereof, and that the agency assess the level of incomplete data 
submitted by hospitals for the APU program to determine the magnitude 
of underreporting, if any, in order to refine how completeness 
assessments may be done in future reporting efforts. In commenting on a 
draft of this report, CMS agreed to implement steps to improve the 
quality and completeness of the data. 

Background: 

Medicare spends over $136 billion annually on inpatient hospital care 
for its beneficiaries. To help ensure the quality of the care it 
purchases through Medicare, CMS launched the Hospital Quality 
Initiative in 2003. This initiative aims to refine and standardize 
hospital data, data transmission, and performance measures as part of 
an effort to stimulate and support significant improvement in the 
quality of hospital care. 

One component of this broader initiative is CMS's participation in the 
Hospital Quality Alliance (HQA), a public-private collaboration that 
seeks to make hospital performance information more accessible to the 
public, payers, and providers of care.[Footnote 14] Before the 
enactment of MMA, HQA had organized a voluntary program for hospitals 
to submit data on quality of care measures intended for public 
reporting. For its part as a participant in HQA, CMS set up a central 
database to receive the data submitted by hospitals and initiated plans 
for a Web site to post information on hospital quality of care 
measures. Thus, CMS had a data collection infrastructure in place when 
MMA established the financial incentive for hospitals to submit quality 
data. 

Selection of Measures: 

The 10 measures chosen by the Secretary of Health and Human Services 
for the APU program are the original 10 measures that were adopted by 
HQA. HQA subsequently adopted additional measures that relate to the 
same three conditions--heart attacks, heart failure, and pneumonia--and 
others that relate to surgical infection prevention. (See table 1 for a 
listing of the APU-measure set and the expanded-measure set.[Footnote 
15]) Hospitals participating in HQA were encouraged to submit data on 
the additional measures, but data submitted on the additional measures 
did not affect whether a hospital received its full payment update 
under the APU program. CMS and the QIOs have tested these measures for 
validity and reliability, and all measures have been endorsed by the 
National Quality Forum, which fosters agreement on national standards 
for measurement and public reporting of health care performance 
data.[Footnote 16] 

Table 1: HQA Hospital Quality Measures: 

APU-measure set: 

For discharges beginning January 1, 2004; 
Heart attack: 
1. Aspirin at arrival; 
2. Aspirin prescribed at discharge; 
3. ACE (angiotensin-converting enzyme) inhibitor for left ventricular 
systolic dysfunction; 
4. Beta blocker at arrival; 
5. Beta blocker prescribed at discharge; 
Heart failure: 
6. Left ventricular function assessment; 
7. ACE inhibitor for left ventricular systolic dysfunction; 
Pneumonia: 
8. Initial antibiotic received within 4 hours of hospital arrival; 
9. Oxygenation assessment; 
10. Pneumococcal vaccination status; 
Surgical infection prevention: (none). 

Expanded-measure set: 

For discharges beginning April 1, 2004; 
Heart attack: 1-5 above plus; 
11. Thrombolytic agent received within 30 minutes of hospital arrival; 
12. PTCA (percutaneous transluminal coronary angioplasty) received 
within 90 minutes of hospital arrival; 
13. Adult smoking cessation advice/counseling; 
Heart failure: 6-7 above plus; 
14. Discharge instructions; 
15. Adult smoking cessation advice/counseling; 
Pneumonia: 
8-10 above plus; 
16. Blood culture performed before first antibiotic received in 
hospital; 
17. Adult smoking cessation advice/counseling; 
Surgical infection prevention: (none). 

For discharges beginning July 1, 2004; 
Heart attack: 1-5, 11-13 above; 
Heart failure: 6-7, 14-15 above; 
Pneumonia: 
8-10, 16-17 above plus; 
18. Initial antibiotic selection for CAP (community-acquired pneumonia) 
in immunocompetent patient; 
19. Influenza vaccination[A]; 
Surgical infection prevention: 
20. Prophylactic antibiotic received within 1 hour prior to surgical 
incision; 
21. Prophylactic antibiotic selection for surgical patients[A]; 
22. Prophylactic antibiotics discontinued within 24 hours after surgery 
end. 

Source: CMS, as of August 4, 2005. 

Note: Measures are worded as CMS posted them on www.qnetexchange.org. 

[A] Hospitals are collecting data for these measures, but public 
reporting of hospital performance on these measures has been postponed. 

[End of table] 

To minimize the data collection burden on hospitals by the APU program, 
CMS and the Joint Commission on Accreditation of Healthcare 
Organizations (JCAHO) have worked to align their procedures and 
protocols for collecting and reporting the specific clinical 
information that is used to score hospitals on the measures. JCAHO-
accredited hospitals--approximately 82 percent of hospitals that 
participate in Medicare--have since 2002 submitted data to JCAHO on the 
same measures as those in the APU-measure set as well as many of those 
in the expanded-measure set. Beginning with the first calendar quarter 
of data submitted by hospitals for the APU program, hospitals had the 
option of submitting the same data to CMS that many of them were 
already collecting for JCAHO. In November 2004, CMS and JCAHO jointly 
issued a manual laying out the aligned procedures and protocols for 
discharges beginning January 1, 2005. 

Collection, Submission, and Reporting of Quality Data: 

Hospitals use CMS's definition of the eligible patient population to 
identify the patients for whom they should collect and submit quality 
data for each measure. The definition is based on the primary diagnosis 
and, for the two cardiac conditions, the age of the patient.[Footnote 
17] Specifically, hospitals use diagnostic codes and demographic 
information from the patients' medical and administrative records to 
determine eligibility based on protocols established by CMS. 

Once the eligible patients have been identified, hospitals extract from 
their patients' medical records the specific data items needed for the 
Iowa Foundation for Medical Care (IFMC) to calculate a hospital's 
performance, following detailed data abstraction guidelines developed 
by CMS. Hospitals may submit data for all eligible patients for a given 
condition, or if they have more than a specified number of eligible 
patients, they may draw a random sample according to a 
formula,[Footnote 18] and submit data for those patients only. These 
data are put into a standardized data format and submitted quarterly 
through a secure Internet connection to the QIO "clinical warehouse" 
administered by IFMC. IFMC accepts into the clinical warehouse only the 
data that meet the formatting and other specifications established by 
CMS[Footnote 19] and that are submitted before the specified deadline 
for that quarter. About 80 percent of hospitals rely on data vendors--
which typically are collecting the same data for JCAHO--to submit the 
data for them. 

IFMC aggregates the information from the individual patient records to 
generate a rate for each hospital on each of the measures for which the 
hospital submitted relevant clinical data. These rates show how often a 
hospital provided the specific service or activity designated in the 
measures to patients for whom that service or activity was appropriate. 
Hospitals also collect information on each patient that identifies 
patients for whom the particular service or activity would not be 
called for, such as patients with a condition that would make 
prescribing aspirin or beta blockers medically inappropriate. 

CMS posts on its Hospital Compare Web site each hospital's rates for 
all the APU and expanded measures for which it submitted data.[Footnote 
20] In November 2004, CMS first posted these rates, based on data from 
the first quarter of calendar year 2004. It subsequently posted new 
rates in March 2005, based on the first two quarters of calendar year 
2004 data, and again in September and December 2005 with additional 
quarters of data. CMS continues to update these rates quarterly, using 
the four most recent quarters of data available. There can be up to a 
14-month time lag between when patients are treated by the hospital and 
when the resulting rates are posted on the CMS Web site. (See fig. 1.): 

Figure 1: Approximate Times for Collection, Submission, and Reporting 
of Hospital Quality Data: 

[See PDF for image] 

[A] CMS had to make its determination of hospital eligibility for the 
fiscal year 2005 annual payment update decision approximately 1 month 
after hospitals submitted their data for the first quarter. 

[End of figure] 

Implementation of the APU Program: 

In implementing the APU program, CMS uses the same policies and 
procedures for collecting and submitting quality data as are used for 
HQA. For the first annual payment update determined by the APU program, 
which applied to fiscal year 2005, hospitals were required to begin 
submitting data by July 1, 2004, for the patients discharged during the 
first calendar quarter of 2004 (January through March 2004). Data were 
received from 3,839 hospitals, over 98 percent of those affected by the 
MMA provision. These figures include 150 hospitals that certified to 
CMS that they had no eligible patients with the three conditions during 
the first calendar quarter of 2004. Hospitals that have no eligible 
patients are not penalized and receive the full annual payment update. 
For the second annual payment update determined by the APU program, 
which applied to fiscal year 2006, participating hospitals were 
required to continue to submit data in accordance with the quarterly 
deadlines set by CMS. Failure to meet the requirements of the program 
and qualify for the full annual payment update in one year does not 
affect a hospital's ability to participate in and qualify for the full 
update in the succeeding year. 

CMS has assigned primary responsibility to the 53 QIOs to inform 
hospitals about the APU program's requirements and to provide technical 
assistance to hospitals in meeting those requirements. This includes 
assistance to hospitals in submitting their data to the clinical 
warehouse provided by IFMC. 

Other Reporting Systems: 

There are several organizations that administer reporting systems that 
collect clinical data, some of which also release their data to the 
public. Some of these organizations are in the public sector, such as 
state health departments, and some are in the private sector, such as 
accreditation bodies. Several of these systems have been in existence 
for a number of years, including one for as long as 16 years. 
Hospitals, health plans, nursing homes, and other external 
organizations submit data to these systems on a range of medical 
conditions, which for most of these systems includes at least one 
cardiac condition (e.g., percutaneous coronary intervention, coronary 
artery bypass grafting, heart attack, heart failure). Many of these 
systems make the results of the data they have collected available for 
public use. For example, one public organization has been collecting 
individual, patient-level data on cardiac surgeries from hospitals for 
the past 16 years and creates reports based on the data collected, 
which it subsequently posts on its Web site. Additionally, data 
collected by these reporting systems can also be used for quality 
improvement efforts and to track performance over time. (For more 
background information on other reporting systems, see app. II, table 
3.)  

CMS Has Processes for Checking Data Accuracy but Has No Ongoing Process 
to Check Completeness: 

CMS has processes for ensuring the accuracy of the quality data 
submitted by hospitals for the APU program, but has no ongoing process 
to assess whether hospitals are submitting complete data. To check 
accuracy, IFMC, a CMS contractor, electronically checks the data as 
they are submitted to the clinical warehouse. In addition, CDAC 
independently audits the data submitted by hospitals. Specifically, it 
reabstracts the quality data from medical records for a sample of five 
patients per quarter for each hospital and compares its results to the 
quality data submitted by hospitals. The data are deemed to be accurate 
if there is 80 percent or greater agreement between these two sets of 
results, a standard that hospitals had to meet for the APU-measure set 
to qualify for their full annual payment update for fiscal year 2006. 
To check completeness, CMS has twice compared the number of cases 
submitted by each hospital for the APU program for a given period to 
the number of claims the hospital submitted to Medicare, once for the 
fiscal year 2005 update and once for the fiscal year 2006 update. 
However, these analyses did not address non-Medicare patient records 
and the approach that CMS took in these analyses was not capable of 
detecting incomplete data for all hospitals. CMS has not put in place 
an ongoing process for checking the completeness of the data that 
hospitals submit for the APU program that would provide accurate and 
consistent information for all patients and all hospitals. Moreover, 
CMS has not required hospitals to certify that they submitted data for 
all eligible patients or a representative sample thereof. 

CMS Checks Data Accuracy Electronically and Through an Independent 
Audit: 

CMS employs two processes to check and ensure the accuracy of the 
quality data submitted by hospitals for the APU program. First, at the 
time that data are submitted to the clinical warehouse, IFMC, a CMS 
contractor, electronically checks the data for inconsistencies and 
missing values. The results are shared with hospitals. After the 
allotted time for review and correction of the submissions, no more 
data or corrections may be submitted by hospitals for that quarter. 
These checks are done whether the hospital submits its data directly to 
the warehouse or through a data vendor. 

Second, CDAC conducts quarterly independent audits to verify that the 
data submitted by hospitals to the clinical warehouse accurately 
reflect the information in their patients' medical records.[Footnote 
21] From among all the patient records submitted to the clinical 
warehouse each quarter, CMS randomly selects for CDAC's reabstraction 
five patient records from each participating hospital.[Footnote 22] 
CDAC sends a request for these patients' medical records to the 
hospitals, and they send photocopies of the records to CDAC for 
reabstraction. A CDAC abstractor reviews the medical record, determines 
if or when a specific action occurred--such as the time when a patient 
arrived at the hospital--and records that data field accordingly. Once 
the CDAC reabstraction is complete, the response previously entered 
into that field by the hospital is compared to that entered by the CDAC 
abstractor, and CDAC notes whether the two responses match. If they do 
not match, a second CDAC abstractor reviews the medical record to make 
a final determination. The results of the CDAC reabstraction are sent 
to the clinical warehouse, where the individual data matches and 
mismatches are summed to produce an accuracy score for each hospital. 
The accuracy score represents the overall percentage of agreement 
between data submitted by the hospital and data reabstracted by CDAC 
across all five cases.[Footnote 23] It is based on all the APU and 
expanded measures for which the hospital submitted data.[Footnote 24] 
The score, along with information from CDAC on where the mismatches 
occurred and why, is shared with the hospital and the hospital's local 
QIO. CMS considers hospitals achieving an accuracy score of 80 percent 
or better to have provided accurate data. Hospitals with accuracy 
scores below 80 have the opportunity to appeal their reabstraction 
results.[Footnote 25] 

In applying these processes for the fiscal year 2005 annual payment 
update, CMS did not require hospitals to meet the 80 percent accuracy 
threshold for the 10 APU measures to qualify for the full update. 
Rather, to receive their full payment update, hospitals only had to 
pass the electronic data checking performed when they submitted their 
data to the clinical warehouse for the first calendar quarter of the 
APU program--for discharges that occurred from January 2004 through 
March 2004. Although the accuracy scores were not considered for the 
payment update, CMS calculated an accuracy score for each quarter in 
which the hospital submitted at least six cases to the clinical 
warehouse. Each quarter the accuracy score was based on data for all 
the measures submitted by the hospital in that quarter and was derived 
from five randomly selected patient records. Along with the accuracy 
score, hospitals received information on where mismatches occurred and 
the reasons for the mismatches. 

In contrast to the prior year, CMS applied the 80 percent threshold for 
accuracy as a requirement for hospitals to qualify for their full 
fiscal year 2006 annual payment update.[Footnote 26] IFMC continued to 
check electronically all of the data as they were submitted for each 
quarter and calculated accuracy scores quarterly for each hospital. CMS 
decided to base its payment update decision on the accuracy score that 
hospitals obtained for the third calendar quarter of 2004--for 
discharges that occurred from July 2004 through September 
2004.[Footnote 27] This meant that the payment decision rested on the 
reabstraction results obtained from 5 randomly selected patient 
records. If a hospital met the 80 percent accuracy threshold based on 
all of the quality data it submitted, it received the full payment 
update. However, if a hospital failed to meet the 80 percent threshold, 
CMS recomputed the accuracy score using only the data elements required 
for the APU-measure set. For hospitals that failed again, CMS combined 
the CDAC reabstraction results from the third calendar quarter of 2004 
with the CDAC results from the fourth calendar quarter of 2004 to 
produce an accuracy score derived from 10 patient medical 
records.[Footnote 28] CMS then computed accuracy scores first for all 
the quality data submitted by the hospital and finally for the APU-
measure set, if needed to reach the 80 percent threshold. As a result, 
even though CMS assessed hospital accuracy primarily on the basis of 
data that exceeded those required for the APU-measure set, hospitals 
were not denied the full annual payment update except on the basis of 
the APU-measure set. A possibility does exist, however, that a hospital 
could have qualified for the full update based on its results for all 
the data it submitted, even if it would have failed using the APU-
measure set. This could happen if the hospital submitted data that 
matched the CDAC abstractors' entries more consistently for the data 
entries used exclusively in computing the expanded measures, such as 
those relating to smoking cessation counseling, than for the data 
required by the APU-measure set. 

In the future, CMS intends to base its decisions on hospital 
eligibility for full annual payment updates on accuracy assessments 
from more than one quarter. Although its concerns about potential 
alignment issues affecting data for the first two quarters of the APU 
program led the agency to rely primarily on data from the third 
calendar quarter for the fiscal year 2006 update, CMS stated that its 
goal was to use accuracy assessments from four consecutive quarters 
when it determines hospital eligibility for the fiscal year 2007 full 
annual payment update. 

CMS uses the accuracy scores in making decisions on payment updates, 
but the scores do not affect the information posted on the Hospital 
Compare Web site. The Web site transmits to the public the rates on the 
APU and expanded measures that derive from the data that the hospitals 
submitted to the clinical warehouse. CMS does not post the accuracy 
scores generated from the CDAC reabstraction process on the Web site or 
indicate if the hospital rates are based on data that met CMS's 80 
percent threshold for accuracy.[Footnote 29] 

CMS Has No Ongoing Process to Ensure Completeness of Data Submitted for 
the APU Program: 

Although CMS has recognized the importance of obtaining quality data 
for the APU program on all eligible patients, or a representative 
sample if appropriate, it has not put in place an ongoing process to 
ensure that this occurs. For the fiscal year 2005 annual payment 
update, CMS checked that hospitals submitted data for at least a 
minimum number of patients by using Medicare claims data to estimate 
the number of "expected cases" that each hospital should have submitted 
to the clinical warehouse. To do this, it first calculated the average 
number of patients for each of the three conditions that each hospital 
had billed Medicare for over the previous eight calendar quarters 
(January 2002 through December 2003). Then, if the average number of 
Medicare claims for a condition was large enough to entitle the 
hospital to draw a sample instead of submitting data for all the 
eligible patients to the clinical warehouse, CMS reduced the number of 
"expected cases" based on the size of the sample.[Footnote 30] CMS told 
each hospital what its expected numbers of heart attack, heart failure, 
and pneumonia patients were. If the actual number of patients for whom 
hospitals submitted data for the APU program was lower, the hospitals 
were instructed to send a letter to their local QIO, signed by the 
hospital's CEO or administrator, stating that the hospital had fewer 
discharged patients for that condition than CMS had estimated. If such 
a letter was filed, the hospital qualified for the full annual payment 
update. In the end, no hospital participating in the APU program was 
denied a full annual payment update for fiscal year 2005 for submitting 
data on an insufficient number of patients or any other reason. 

For the fiscal year 2006 update decision, CMS took a different approach 
to using Medicare claims data to address the issue of completeness. CMS 
used Medicare claims data to check whether hospitals that billed 
Medicare for any cases with one of the three conditions submitted at 
least one case to the clinical warehouse. To do this, CMS compared each 
hospital's Medicare claims for the three conditions for the four 
calendar quarters of 2004 to the hospital's submissions to the clinical 
warehouse for those same quarters. CMS identified instances where 
hospitals had submitted one or more claims for payment to Medicare for 
any of the three conditions for a quarter when they had not submitted 
any cases with one of those conditions to the clinical warehouse. On 
this basis, CMS determined that 110 hospitals would not qualify for the 
full payment update for fiscal year 2006. 

CMS conducted two additional analyses involving a comparison of the 
same Medicare claims data and quality data submissions to identify 
hospitals that may have submitted incomplete data for the APU program, 
but these analyses did not affect hospital eligibility for the full 
fiscal year 2006 payment update. The additional analyses identified (1) 
a set of hospitals that may have submitted samples of their eligible 
cases to the clinical warehouse when, according to the applicable 
sampling rules, they should have submitted data on all their cases; and 
(2) another set of hospitals that failed to submit cases to the 
clinical warehouse for all of the three conditions for which they filed 
Medicare claims in that quarter. However, in contrast to the hospitals 
that did not qualify for their full payment update, the hospitals in 
the second set submitted to the clinical warehouse at least one case 
for one of the three conditions. A CMS official stated that the agency 
plans to educate the hospitals identified by these additional analyses 
on the data submission and sampling requirements for the APU program. 

The analysis that CMS conducted using Medicare claims data for its 
fiscal year 2005 update decision and the three analyses it conducted in 
conjunction with its fiscal year 2006 update decision shared two 
limitations: none addressed the completeness of data submissions for 
non-Medicare patients, and none could detect incomplete data for all 
hospitals. Given that non-Medicare patients represent a substantial 
proportion of the patients treated for heart attacks, heart failure, 
and pneumonia,[Footnote 31] any minimum number of "expected cases" 
based on Medicare claims inherently underestimates the total number of 
patients for which hospitals should have submitted quality data for the 
APU program. Moreover, the approaches taken in the analyses conducted 
for both fiscal year updates could not detect incomplete data for many 
hospitals. For example, in the fiscal year 2005 analysis, the 
difference between the number of cases expected under the CMS sampling 
rules and the higher number expected under the sampling rules that 
applied to JCAHO-accredited hospitals meant that JCAHO-accredited 
hospitals treating more patients than the minimum CMS sample of seven 
could have failed to submit data on most of the cases that exceeded the 
CMS minimum and still have met the number of expected cases set by 
CMS.[Footnote 32] The analysis that CMS conducted to determine hospital 
eligibility for the full fiscal year 2006 update also could identify 
only certain hospitals that submitted incomplete data, in this case 
limited to hospitals that submitted no patient data at all to the 
clinical warehouse in a given quarter. 

CMS officials acknowledged that the lack of information on non-Medicare 
patients and the imprecise adjustments that CMS made to take account of 
the varying sampling procedures that hospitals could have followed 
limited the conclusions that CMS could draw from its Medicare claims 
data analysis for the fiscal year 2005 update. Because of these 
limitations, CMS officials described their effort as a rough check for 
inconsistencies between data submitted by hospitals to the clinical 
warehouse and the cases that the hospitals had billed to Medicare. 

CMS has not combined these limited efforts to monitor the completeness 
of hospital quality data submissions with efforts to clearly inform 
hospital officials of their obligation to submit complete data. For 
example, CMS has not explicitly listed submission of complete data as a 
requirement for participating in the APU program on the "Notice of 
Participation" that the hospital CEO or administrator must sign when 
hospitals enroll. The notice states requirements for participating 
hospitals--including that they must register with the QualityNet 
Exchange Web site[Footnote 33] and that they must submit data for all 
measures specified in the APU-measure set by established deadlines. The 
notice indicates that the submitted data will undergo validation, a 
reference to the CDAC reabstraction process. However, the notice does 
not stipulate that hospitals must submit data for all eligible cases, 
or for a representative sample if appropriate. 

We interviewed health professionals familiar with the APU program, 
several of whom raised concerns about data completeness. One expert in 
the area of outcomes research noted the potential for systematic 
underreporting by hospitals. He suggested that, as one approach to 
detect systematic underreporting, CMS could compare not only the number 
of patients for whom data were submitted and Medicare claims filed, but 
also the characteristics of patients for cases submitted to the APU 
program to the patient characteristics of comparable cases submitted to 
Medicare for payment. Another expert in the area of clinical quality 
improvement expressed his concern that the APU program did not verify 
the completeness of the data. He observed that hospitals have 
flexibility in determining which patients are included through their 
assignment of the patient's primary diagnosis. A QIO official echoed 
this concern, noting the risk that hospitals could decide to not submit 
cases where patients had not received the services or activities 
assessed by the APU measures. 

Data Accuracy Baseline Was High Overall, but Statistically Uncertain 
for Many Hospitals, and Data Completeness Baseline Cannot Be 
Determined: 

We could determine a baseline level of accuracy for the quality data 
submitted for the APU program but not a baseline level of completeness. 
We found a high overall baseline level of accuracy when we examined 
CMS's assessment of the data submitted by hospitals for the first two 
calendar quarters of 2004. The median accuracy score exceeded 90 
percent, which was well above the 80 percent accuracy threshold set by 
CMS, and about 90 percent of hospitals met or exceeded that threshold 
for both the first and the second calendar quarters of 2004. For most 
hospitals whose accuracy scores were well above the threshold, the 
results were statistically certain. However, for approximately one-
fourth to one-third of all the hospitals that CMS assessed for 
accuracy, the statistical margin of error for their accuracy score 
included both passing and failing accuracy levels. Consequently, for 
these hospitals, the small number of cases that CMS examined was not 
sufficient to establish with statistical certainty whether the hospital 
met the threshold level of data accuracy. Accuracy did not vary between 
rural and urban hospitals, and small hospitals provided data as 
accurate as those from larger hospitals. The completeness baseline 
could not be determined because CMS did not assess the extent to which 
all hospitals submitted data on all eligible patients, or a 
representative sample thereof, for the first two calendar quarters of 
2004, and consequently there were no data from which to derive such an 
assessment. 

Baseline Level of Data Accuracy Was High Overall, and Large Majority of 
Hospitals Met Accuracy Threshold: 

Overall, the baseline level of data accuracy for the first two quarters 
of the APU program was high. The median accuracy score achieved by 
hospitals ranged between 90 and 94 percent, with slightly higher values 
in the second quarter and for the APU-measure set. (See fig. 2.) In 
addition, with at least half the hospitals receiving accuracy scores 
above 90, relatively few failed to reach the 80 percent threshold set 
by CMS. 

Figure 2: Baseline Hospital Accuracy Scores at Selected Percentiles, by 
Measure Set and Quarter: 

[See PDF for image] 

Note: Figure reflects accuracy scores for hospitals covered by the APU 
program. Hospitals that submitted fewer than six cases to the clinical 
warehouse in a quarter did not undergo CDAC reabstraction and therefore 
did not receive an accuracy score for that quarter. Calculation of 
accuracy scores for the expanded-measure set was based on all the 
measures for which a hospital submitted data, which could range from 
the APU measures alone to a maximum of 17--the APU measures plus as 
many as 7 additional measures. 

[End of figure] 

In both quarters, 90 to 92 percent of hospitals obtained accuracy 
scores meeting the threshold using the APU-measure set, and 87 to 90 
percent met the threshold using the expanded-measure set (see table 
2).[Footnote 34] The 8 to 13 percent of hospitals that did not meet the 
accuracy threshold represented approximately 300 to 500 hospitals 
across the country. 

Table 2: Percentage and Number of Hospitals Whose Baseline Accuracy 
Score Met or Fell Below the 80 Percent Threshold, by Measure Set and 
Quarter: 

[See PDF for image] 

Source: GAO analysis of CMS data. 

Note: Calculation of accuracy scores for the expanded-measure set was 
based on all the measures for which a hospital submitted data, which 
could range from the APU measures alone to a maximum of 17--the APU 
measures plus as many as 7 additional measures. 

[End of table] 

There were minimal differences in baseline accuracy scores among 
hospitals characterized by urban or rural location and small or large 
capacity,[Footnote 35] but variation across hospitals served by 
different data vendors was more substantial. Rural hospitals and 
smaller hospitals generally received accuracy scores similar to those 
of urban hospitals and larger hospitals.[Footnote 36] Among the 
hospitals that used JCAHO-certified data vendors to submit their 
quality data to the clinical warehouse, a higher percentage of 
hospitals served by certain data vendors met the 80 percent threshold 
than did the hospitals served by other data vendors (see app. III, 
table 8).[Footnote 37] 

Passing the 80 Percent Threshold Is Statistically Uncertain for One-
Fourth to One-Third of Hospitals: 

While the baseline level of data accuracy achieved by hospitals in the 
aggregate was well above the 80 percent threshold, for approximately 
one-fourth to one-third of hospitals the determination that a 
particular hospital met the 80 percent threshold was statistically 
uncertain. This uncertainty stems primarily from the small number of 
cases examined for accuracy from each hospital. Because CDAC's 
reabstraction of the data is limited to five patient records per 
quarter, the greater sampling variability found in small samples leads 
to relatively large confidence intervals, reflecting low statistical 
precision, for the accuracy score of any specific hospital.[Footnote 
38] Across all hospitals, the median difference between the upper and 
lower limits of the confidence interval was 14.0 percentage points 
using the APU-measure set for first-quarter discharges, dropping to 
11.8 percentage points in the second quarter.[Footnote 39] For the 
expanded-measure set, the median confidence interval was 14.6 
percentage points in the first quarter and 13.0 percentage points in 
the second. 

The wide confidence intervals meant that for a substantial number of 
hospitals it was statistically uncertain whether a different sample of 
cases would have altered their result from passing the 80 percent 
threshold to failing, or vice versa.[Footnote 40] For most hospitals 
there was statistical certainty that their baseline accuracy score met 
CMS's 80 percent accuracy threshold. However, other hospitals had 
confidence intervals for their accuracy scores where the upper limit 
was 80 or above and the lower limit was less than 80. Because the 
confidence interval around the accuracy score computed for each of 
these hospitals bracketed the accuracy threshold set by CMS, their 
results were statistically uncertain.[Footnote 41] Consequently, for 
these hospitals, the small number of cases that CMS examined was not 
sufficient to establish whether the hospital met the threshold level 
for data accuracy. One-third of all the hospitals that CMS assessed for 
accuracy fell into this uncertain category for first-quarter 2004 
discharges using the APU-measure set. (See fig. 3.) This proportion 
declined to about one-fourth of the hospitals for the second quarter. 
When the expanded-measure set was used--as CMS has done when 
calculating its quarterly accuracy scores--the proportion of hospitals 
whose accuracy scores were statistically uncertain increased compared 
to the APU-measure set for both the first and the second quarter. 

Figure 3: Percentage of Hospitals Whose Baseline Accuracy Score 
Confidence Intervals Clearly Exceed, Fall Below, or Include the 80 
Percent Threshold, by Measure Set and Quarter: 

[See PDF for image] 

Note: The confidence interval is based on a 95 percent significance 
level. Calculation of the accuracy scores and confidence intervals for 
the expanded-measure set was based on all the measures for which a 
hospital submitted data, which could range from the APU measures alone 
to a maximum of 17--the APU measures plus as many as 7 additional 
measures. 

[End of figure] 

These confidence intervals would narrow if CMS drew on multiple 
quarters of data to bring more cases into the computation of the 
accuracy scores. CMS has stated its intention to base this accuracy 
assessment on four quarters of hospital quality data, but so far every 
accuracy score it has generated and reported to hospitals has been 
based on a single quarter of data. Moreover, its implementation of the 
fiscal year 2006 payment update called for using only one quarter of 
data, with the possibility of adding one more quarter of data for 
hospitals that failed to meet the accuracy threshold based on the 
single quarter of data.[Footnote 42] 

No Data Were Available to Provide Baseline Assessment of Completeness 
of Hospital Quality Data: 

There were no data available from which to estimate a baseline level of 
completeness for the first two calendar quarters of data submitted for 
the APU program. In contrast to the system of quarterly reabstractions 
performed by CDAC to check the accuracy of quality data submitted by 
hospitals, CMS did not conduct any corresponding assessment of the 
extent to which all hospitals submitted data on all the cases, or a 
representative sample of such cases, that met CMS's eligibility 
criteria for the first two calendar quarters of 2004. 

The information that CMS did collect was not suitable for estimating 
the baseline level of data completeness. The Medicare claims data 
analysis conducted by CMS on the first calendar quarter of data 
submitted for the APU program was not designed to provide valid 
information on the magnitude of data incompleteness for each hospital, 
which is what is needed to estimate a baseline level of data 
completeness. Although CMS could identify instances where certain 
hospitals failed to provide quality data on all eligible cases, CMS's 
analysis did not produce comparable information on data completeness 
for every hospital. As noted above, it lacked information on non-
Medicare patients and could not adjust properly for the sample sizes 
that JCAHO-accredited hospitals would have drawn if they followed 
JCAHO's sampling rules rather than CMS's. The limitations in the CMS 
analysis would affect some hospitals more than others, depending on how 
many non-Medicare patients a hospital treated and whether it applied 
the JCAHO sampling rules. Consequently, had we used information from 
this analysis to estimate baseline data completeness, our results would 
have been distorted by the uneven impact of those factors on the 
information produced for different hospitals.[Footnote 43] 

In addition, we found no data for assessing the baseline completeness 
of the quality data provided by hospitals submitting samples of their 
eligible cases to the clinical warehouse. For hospitals that submitted 
a sample, their quality data could be incomplete, even if they 
submitted the expected number of cases, if their samples were not 
selected in a way that ensured they were representative of all a 
hospital's patients. If a hospital did not follow appropriate 
procedures to provide for random selection, the sample might not be 
representative and therefore could be incomplete. Because the available 
information from CMS focused on the number of cases submitted, and not 
on how they were selected, we could not address this aspect of data 
completeness. 

Other Reporting Systems Use Various Methods to Ensure Data Accuracy and 
Completeness, Notably an Independent Audit: 

Other reporting systems that collect clinical performance data have 
adopted various methods to ensure data accuracy and completeness, and 
officials from these systems stressed the importance of including an 
independent audit in these activities. Most other reporting systems 
that conduct independent audits incorporate three methods as part of 
their process that CMS does not use in its independent audit. 
Specifically, these systems include an on-site visit, focus their audit 
on a selected number of facilities or reporting entities, and review a 
minimum of 50 patient medical records per reporting entity. 

Other Reporting Systems Use Various Methods to Check Data: 

Other reporting systems that collect clinical performance data have 
adopted various methods to ensure data accuracy and completeness. To 
check data accuracy, all the other reporting systems we examined assess 
the data when they are submitted, typically using computers to detect 
missing or out-of-range data. (See app. II, tables 4 and 5.) In 
addition, all the other systems have developed standardized data 
collection processes and measures. When checking data completeness, all 
the other systems compare submitted data with data from another source, 
whether inside the facility, such as pharmacy or laboratory records, or 
outside the facility, such as state hospital discharge data or Medicare 
claims data. Officials reported that these analyses were done annually 
or had been done one time, and one said that additional studies were 
planned.[Footnote 44] Officials from these systems also cite various 
other methods to consider when ensuring data accuracy and completeness, 
including reviewing established measures annually, identifying a point 
person at each facility to provide consistency, establishing channels 
for ongoing communication, and providing training on a continuous 
basis.[Footnote 45] 

Other Reporting Systems Conduct Independent Audits: 

Most other reporting system officials we interviewed conduct 
independent audits that include a comparison of submitted data to 
medical records. Most other reporting systems that conduct independent 
audits incorporate three methods as part of their process that CMS does 
not use in its independent audit. Specifically, they (1) include an on-
site visit as part of their independent audit, (2) focus their audits 
on a selected number of facilities or reporting entities, and (3) 
review a minimum of 50 patient medical records per reporting entity 
during the auditing process. During an on-site visit, auditors are able 
to review patient medical records for accuracy and interview staff when 
additional information is needed. Auditors are also able to check the 
data submitted to their system against other data sources at the 
facilities, including physician notes, patient or resident rosters, 
billing records, laboratory records, and pharmacy records. In addition, 
because auditors from other reporting systems may not visit every 
facility,[Footnote 46] the systems use various methods to focus the 
auditing process when selecting which facilities to visit. These 
include auditing a percentage of all eligible facilities, auditing 
facilities that did particularly well or poorly, and auditing a subset 
of facilities each year. Furthermore, most of the other reporting 
systems that conduct independent audits review a minimum of 50 patient 
medical records per audited entity as part of their independent 
auditing process. When selecting which patient medical records to 
review, some systems take a random sample of the patient population, 
one system reviews all deaths at the selected facility, and another 
reviews all instances where the patient died from shock as a result of 
percutaneous coronary intervention. 

Officials at other reporting systems we interviewed and an expert in 
the field stressed the importance of the independent audit. For 
example, an official from one of the other reporting systems said that 
audits conducted by an independent third party are "the best way" to 
ensure data accuracy and completeness. An official from another 
reporting system said that having someone independently check the data 
is "one of the most important things" that an organization can do to 
check data accuracy and completeness. Additionally, an expert we 
interviewed said that independent, external audits are "essential." 
Though most of the other reporting systems employ an independent 
auditing process, officials from one system that has yet to implement 
such a process said their organization recognizes the importance of 
independently checking the data and is currently designing and 
implementing an independent auditing process. 

Conclusions: 

Data collected for the APU program affect the payment received by 
hospitals from Medicare and are used to inform the public about 
hospital quality. For both these purposes, it is important that CMS is 
able to ensure that the data are reliable in terms of both accuracy and 
completeness. 

CMS has put in place an ongoing process for assessing the accuracy of 
quality data submitted by hospitals, but the process has limitations. 
Although CMS checks the accuracy of data electronically as they are 
submitted and through an independent audit conducted by CDAC, the 
latter process is limited by the selection of only five cases per 
quarter per hospital, regardless of the hospital's size. Most hospitals 
had high baseline accuracy scores that were statistically certain. 
However, for about one-fourth to one-third of all the hospitals that 
CMS assessed for the first two calendar quarters of 2004, CMS's 
determination as to whether the hospital met its accuracy standard was 
statistically uncertain. This was due primarily to the small number of 
cases selected for an audit. Although CMS has stated its intention to 
look at more cases by pooling reabstraction results from more than one 
calendar quarter, all of the hospital accuracy reports that it has 
generated to date have been based on a single quarter of data. 
Officials from other reporting systems that collect clinical 
performance data told us that they also use an independent audit to 
check data accuracy, but generally sample a larger number of patient 
medical records, either by sampling a percentage of total cases 
submitted or by identifying a minimum number of cases in the sample. In 
addition, most other reporting systems focused their audits on a 
selected number of facilities. 

In contrast to CMS's establishment of an ongoing process for assessing 
data accuracy, the agency has not put in place an ongoing process to 
check the completeness of the data that hospitals submit. Because of 
the purposes for which these data may be used, there could be an 
incentive for hospitals to selectively report data on cases that score 
well on the quality measures. With no ongoing way to check 
completeness, CMS does not know whether or how often hospitals submit 
incomplete data. We believe this is a significant gap in oversight. The 
process used for the fiscal year 2005 annual payment update compared 
hospital submissions to Medicare claims data, but as CMS has noted, 
this did not provide a comparable assessment of each hospital's data, 
even for Medicare patients alone. Moreover, in its comparison of 
hospital quality data submissions with Medicare claims for the fiscal 
year 2006 update, CMS identified more than 100 hospitals that had 
treated eligible patients in a given quarter but had not submitted data 
on a single case for that quarter to the clinical warehouse. Yet CMS 
has not asked hospitals to certify that the data they have submitted 
constitute all, or a representative sample, of the eligible patient 
population. The various methods used by other reporting systems to 
check the completeness of data illustrate the variety of approaches 
that are available. These include conducting on-site visits as part of 
their independent audit, comparing data submissions to data from 
another source maintained by the facility or external to it, and 
performing such checks annually or planned at specified intervals. 

Given CMS's plans to continue public reporting efforts after the APU 
program ends, we believe that processes for checking the reliability of 
data should continue to be refined in order for the individuals and 
organizations that use the data to have confidence in the information. 

Recommendations for Executive Action: 

In order for CMS to help ensure the reliability of the quality data it 
uses to produce information on hospital performance, we recommend that 
the CMS Administrator undertake the following three actions: 

* focusing on the subset of hospitals for which it is statistically 
uncertain if they met CMS's accuracy threshold in one or more previous 
quarters, increase the number of patient records reabstracted by CDAC 
in a subsequent quarter so that the proportion of hospitals with 
statistically uncertain results is reduced; 

* require hospitals to certify that they took steps to ensure that they 
submitted data on all eligible patients, or a representative sample 
thereof; and: 

* assess the level of incomplete data submitted by hospitals for the 
APU program to determine the magnitude of underreporting, if any, in 
order to refine how completeness assessments may be done in future 
reporting efforts. 

Agency Comments: 

In commenting on a draft of this report, CMS stated it appreciated our 
analysis and recommendations. (CMS's comments appear in app. IV.) The 
agency noted that the APU program led to a dramatic increase in the 
number of hospitals that submitted data on the designated 10 quality 
measures, resulting in public reporting of quality data for about 3,600 
hospitals on the agency's Web site. In addition, CMS described the 
steps it had taken to ensure the accuracy and completeness of the 
quality data submitted by hospitals for the APU program. It said that 
the methods it had used were sound, but it agreed that the quality and 
completeness of the data must be improved. 

With respect to reducing the statistical uncertainty of its assessments 
of the accuracy of hospital quality data submissions, CMS agreed that 
the quarterly accuracy assessments based on five patient charts can 
have considerable sampling error and stated that it would improve the 
stability of its accuracy assessments by using data from four calendar 
quarters when it assessed hospital eligibility for the fiscal year 2007 
annual payment update. CMS stated a concern with having sufficient time 
within the current data submission schedule to increase the number of 
patient records reabstracted. However, we recommended in the draft 
report that hospitals with statistically uncertain results in one or 
more previous quarters have an increased number of records 
reabstracted. The assessment of statistical uncertainty for a hospital 
and the reabstraction of additional records do not need to occur within 
the same quarter. We have modified slightly the wording of the 
recommendation to clarify the intended timing of these additional 
reabstractions. 

With respect to ensuring the completeness of quality data submitted by 
hospitals, CMS agreed that it needs to improve its methods. CMS noted 
that its comparison of hospital data quality submissions to the claims 
filed by those hospitals to be paid for treating Medicare beneficiaries 
uncovered numerous discrepancies. The agency agreed with our 
recommendation to require hospitals to formally attest to the 
completeness of the quality data that they submit quarterly. In 
addition, CMS stated that it would also require each hospital to report 
the total number of Medicare and non-Medicare patients who were 
eligible for quality assessment under the APU program. 

In terms of assessing the level of incomplete data for the APU program, 
CMS said it had a process in place to accomplish this, but as we stated 
in the draft report, CMS's process did not cover all patients and all 
hospitals because it lacked information on non-Medicare patients even 
though hospitals were required to submit data on both Medicare and non-
Medicare patients. Additionally, the tests that CMS applied could 
detect incomplete data for only a limited subset of hospitals, in 
contrast to its assessment of data accuracy which covered all hospitals 
that submitted data on six or more cases in a quarter. CMS acknowledged 
it could assess completeness only for Medicare patients, but said that 
by requiring hospitals to report an aggregate count of all eligible 
patients, it would henceforth have the data needed to assess the 
completeness of both Medicare and non-Medicare quality data 
submissions. The agency stated it will use these data to provide 
quarterly feedback to hospitals about the accuracy and completeness of 
their data submissions, and require them to explain discrepancies 
between the data they have submitted for the APU program and the 
aggregate count of eligible patients they have reported. CMS has not 
said that it will determine the magnitude of underreporting for the 
program as a whole, as we recommended. Additionally, by relying on the 
hospitals themselves to supply data on the number of non-Medicare 
patients, CMS's proposed approach lacks an independent verification of 
the completeness of submitted data. This contrasts with the practice of 
most of the other reporting systems we contacted, as well as experts in 
the field, who generally underscored the importance of independently 
checking both the accuracy and the completeness of the quality data. 

As arranged with your offices, unless you publicly announce its 
contents earlier, we plan no further distribution of this report until 
30 days after its issue date. At that time, we will send copies of this 
report to the Administrator of CMS and other interested parties. We 
will also make copies available to others on request. In addition, the 
report will be available at no charge on GAO's Web site at 
http://www.gao.gov. 

If you or your staffs have any questions about this report, please 
contact me at (202) 512-7101 or BascettaC@gao.gov. Contact points for 
our Offices of Congressional Relations and Public Affairs may be found 
on the last page of this report. GAO staff who made major contributions 
to this report are listed in appendix V. 

Cynthia A. Bascetta: 
Director, Health Care: 

[End of section] 

Appendix I: Scope and Methodology: 

To determine the processes used by the Centers for Medicare & Medicaid 
Services (CMS) to ensure the accuracy and completeness of data 
submitted by hospitals for the Annual Payment Update program (APU 
program), we interviewed both CMS officials and staff at DynKePRO--
which operates the Clinical Data Abstraction Center (CDAC)--and the 
Iowa Foundation for Medical Care (IFMC), two contractors that perform 
data collection and data quality monitoring tasks for the APU program. 
In addition, we reviewed documentation on the program available 
publicly on the Quality Net Exchange Web site[Footnote 47] and the Web 
sites of several quality improvement organizations (QIO)--contractors 
to CMS that provide technical assistance to hospitals on the APU 
program--as well as documents on the APU program provided to us at our 
request by CMS. We also obtained access to CMS's intranet system and 
searched for relevant memorandums and other documents regarding CMS's 
policies and requirements for hospitals that participated in the APU 
program. To gain insights from other groups involved in the APU 
program, we interviewed officials from two or more QIOs, state hospital 
associations, and hospital data vendors that submitted data to the IFMC-
operated database for their hospital clients. 

Our assessment of the baseline accuracy of the initial APU program data 
depended on the availability of suitable information from CMS. We 
examined CMS's reabstraction process to determine if the CDAC 
assessments of data accuracy would be appropriate for that purpose. 
Reabstraction is the re-collection of clinical data for the purpose of 
assessing the accuracy of data abstractions performed by hospitals. In 
the APU program, CDAC compares data reported by the hospitals to those 
it has independently obtained from the same medical records. CDAC has 
instituted a range of procedures, including training of its abstractors 
and continuous monitoring of interrater reliability, intended to ensure 
that its abstractors understand and follow its detailed guidance for 
arriving at abstraction determinations that are correct in terms of 
CMS's data specifications. We interviewed CDAC staff and observed the 
implementation of these procedures during a site visit at the CDAC 
facility. On the basis of this information we concluded that it would 
be appropriate for us to use the results of the CDAC reabstractions to 
estimate baseline data accuracy for the APU program. 

We obtained the results of the reabstractions that CDAC had conducted 
on samples of the patients for whom hospitals had submitted data from 
the first two quarters of 2004. These two quarters were the first two 
data submissions made by hospitals under the APU program and the most 
recent available when we conducted these analyses. They constituted 
20,465 patient records for the first quarter and 20,259 for the second. 
These files showed, for each data element that CMS used in assessing 
abstraction accuracy, the correct entry as determined by the CDAC 
abstractors and whether this matched the value that the hospital had 
reported. We applied CMS's algorithms for computing hospital scores on 
the expanded-measure set in order to determine the extent of missing or 
invalid data. We found that approximately 2 to 3 percent of patient 
records could not be scored on any given APU measure due to missing 
data. We excluded from the analysis records from critical access 
hospitals and acute care hospitals in Maryland and Puerto Rico (which 
are paid under different payment systems than other acute care 
hospitals and therefore are not subject to a reduced annual payment 
update under the APU program[Footnote 48]) and a small number of 
records not related to the three medical conditions covered by the APU 
program.[Footnote 49] 

Next we applied the scoring rules developed by CMS to assess the 
accuracy of hospital abstractions. We calculated the accuracy score for 
each hospital in each quarter, using the data elements needed for the 
APU-measure set and, separately, for the expanded-measure set. Accuracy 
scores for the expanded-measure set are based on all the measures for 
which a hospital submitted data, which could range from the APU 
measures alone to a maximum of 17--the 10 measures in the APU-measure 
set plus the 7 additional measures adopted by the Hospital Quality 
Alliance for hospital discharges through the second calendar quarter of 
2004. These scores represented the proportion of data elements where 
CDAC and the hospital agreed, summing across all the assessed data 
elements for the five sampled cases. We then calculated the 
distribution of those scores, and the proportion of hospitals that met 
or exceeded the 80 accuracy threshold that CMS had set. Next we 
calculated the confidence interval for each of those accuracy scores, 
using the formula that CMS had selected for that purpose. However, 
whereas CMS applied a one-tailed test--passing any hospital that had a 
confidence interval whose upper bound reached 80 or above--we applied a 
two-tailed test to assess the statistical uncertainty attached to both 
passing and failing the threshold. The one-tailed test that CMS applied 
prevented hospitals from losing their full annual payment update on the 
basis of their accuracy score if there was less than a 95 percent 
probability that a score below 80 would have remained below 80 in 
another sample. This meant that hospitals with large confidence 
intervals could have accuracy scores well below 80 and still pass the 
CMS accuracy requirement. Our analysis focused instead on assessing the 
level of statistical certainty for all the accuracy scores, both above 
and below the 80 percent threshold. We sought to identify passing as 
well as failing scores that could have changed with another sample. To 
do so, we applied a two-tailed test and observed whether a hospital's 
confidence interval bracketed the 80 percent threshold. 

To provide descriptive information about variation in the accuracy 
scores obtained by hospitals in different situations, we collected 
additional information about the hospitals from other sources. From the 
Medicare Provider of Services file we obtained the Social Security 
Administration metropolitan statistical area code (referred to as the 
SSA MSA code) and Social Security Administration metropolitan 
statistical area size code (referred to as the SSA MSA size code) to 
distinguish between urban and rural hospitals. We also obtained from 
that source the total number of Medicare-certified beds in order to 
categorize hospitals by size. To compare the accuracy scores of 
hospitals that employed different data vendors, we obtained from IFMC 
the identification codes (but not the names) of the various data 
vendors certified by the Joint Commission on Accreditation of 
Healthcare Organizations (JCAHO) that had submitted to the clinical 
warehouse data for the APU program on behalf of hospitals they served. 
Those codes were also available in the case tracking information for 
the patient records in the CDAC database. We then identified for each 
CDAC reabstraction whether the case had originally been submitted by a 
JCAHO-certified data vendor, and if so, which one. These data were 
aggregated to generate accuracy scores for each hospital that 
consistently submitted its quality data through one data vendor in a 
given quarter. This allowed us to determine the proportion of hospitals 
served by each JCAHO data vendor that met CMS's 80 percent accuracy 
threshold. We also calculated the proportion of hospitals that 
submitted their own quality data to CMS (identified in the CDAC case 
tracking information by the hospital's Medicare provider ID number) 
that met the accuracy threshold. Although this analysis was limited to 
data vendors that were JCAHO-certified, those vendors collectively 
submitted data to the clinical warehouse for 78 to 79 percent of the 
hospitals we analyzed in the two baseline quarters. Another 13 to 14 
percent of hospitals directly submitted their own data, and we do not 
have information on how the remaining hospitals submitted data to the 
clinical warehouse. 

As was the case for our baseline accuracy assessment, our assessment of 
the baseline completeness of the data submitted for the APU program 
depended on the availability of suitable data from CMS. Specifically, 
we considered using CMS's estimates of minimum expected cases derived 
from Medicare claims data to arrive at estimates of baseline 
completeness. The CMS officials we spoke with noted that there were 
numerous reasons why the two data sources--quality data submissions for 
the APU program and cases billed to Medicare--would be expected to 
diverge, apart from any underreporting of quality data by hospitals. 
The claims data were limited to Medicare fee-for-service patients, 
whereas the hospitals were obliged to submit quality data on all 
patients over 18 years of age (over 28 days old for most pneumonia 
measures), including patients belonging to Medicare health maintenance 
organizations. In addition, hospitals with large numbers of cases could 
draw samples for the quality data, but would bill for all patients. In 
making adjustments to its number of "expected cases" to take account of 
sampling, CMS found that it could not reliably identify the hospitals 
that should have followed the JCAHO sampling rules, which would result 
in larger-sized samples. Therefore, in calculating the number of cases 
it expected hospitals to have submitted to the clinical warehouse, CMS 
applied to all hospitals across the board the expectation of smaller 
samples based on rules that pertained to hospitals not accredited by 
JCAHO. Finally, the Medicare data used for the comparison was an 
average volume recorded over the previous 2 years, not claims filed for 
the quarter to which the quality data applied. 

We found that these limitations precluded our using information from 
CMS's Medicare claims analysis to assess the baseline completeness of 
the data submitted by hospitals for the APU program. CMS's comparison 
of hospital quality data submissions to the clinical warehouse to its 
estimated number of "expected cases" might have served CMS's purposes, 
by identifying at least some instances of significant discrepancy 
between the number of cases for which quality data were submitted and 
claims filed. However, we determined that it would not provide a 
reasonable estimate of the magnitude of data completeness for all 
hospitals. Because the limitations in the CMS analysis would affect 
some hospitals more than others, depending on how many non-Medicare 
patients a hospital treated and whether it applied the JCAHO sampling 
rules, we concluded that using information from this analysis to 
estimate baseline data completeness would lead to results that were 
distorted by the uneven impact of those factors on the information 
produced for different hospitals. 

To obtain information on other processes that could be used to check 
data accuracy and completeness, we interviewed officials from 
organizations that administer reporting systems that collect clinical 
performance data. To select these organizations, we took several steps. 
We reviewed reports on reporting systems, including two issued by QIOs: 
IPRO's 2003 Review of Hospital Quality Reports and Delmarva 
Foundation's The State-of-the-Art of Online Hospital Public Reporting: 
A Review of Forty-Seven Websites.[Footnote 50] We solicited input from 
the authors of each report and interviewed academic researchers who 
have researched methods of assessing the reliability of performance 
data. We used on-line resources to obtain information on federal-and 
state-administered surveillance efforts. Our selection criteria focused 
on systems that collected clinical data, as opposed to administrative 
or claims data, and that were mentioned most often in the reports and 
interviews cited above. To ensure variation, we selected a mix of 
systems, including those run by public and private organizations, those 
receiving data from hospitals and those receiving data from other types 
of providers, and those collecting data across a range of medical 
conditions and those collecting data on specific medical conditions. 
Using a structured protocol, we interviewed officials from the 
following organizations: JCAHO, National Committee for Quality 
Assurance, Society of Thoracic Surgeons, California Office of Statewide 
Health Planning and Development, New York State Department of Health, 
CMS (the units responsible for monitoring nursing home care regarding 
the Data Assessment and Verification Project (DAVE) contract), and the 
American College of Cardiology. Each organization reviewed and 
confirmed the accuracy of the information presented in appendix II. 

Our analysis is based on the quality measures established for the APU 
program and the information available as of September 2005 on the 
accuracy and completeness of data submitted by hospitals for that 
program. We did not evaluate the appropriateness of these quality 
measures relative to others that could have been selected. Nor did we 
examine the actual performance by hospitals on the measures (e.g., how 
often they provide a particular service or treatment). Our analysis of 
the baseline level of accuracy and completeness of data submitted for 
the APU program is based on the procedures developed by CMS to validate 
the data submitted. We have not independently compared the data 
submitted by hospitals to the original patient clinical records. 

We conducted our work from November 2004 through January 2006 in 
accordance with generally accepted government auditing standards. 

[End of section] 

Appendix II: Other Reporting Systems: 

Table 3: Background Information on CMS and Other Reporting Systems: 

Organization status; 
Centers for Medicare & Medicaid Services (CMS): Public; 
Other reporting systems: American College of Cardiology (ACC): Private, 
nonprofit; 
Other reporting systems: California Office of Statewide Health Planning 
and Development: Public; 
Other reporting systems: Data Assessment and Verification Project 
(DAVE)[A]: Public; 
Other reporting systems: Joint Commission on Accreditation of 
Healthcare Organizations (JCAHO)[B]: Private, nonprofit; 
Other reporting systems: National Committee for Quality Assurance 
(NCQA): Private, nonprofit; 
Other reporting systems: New York State Department of Health: Public; 
Other reporting systems: Society of Thoracic Surgeons (STS): Private, 
nonprofit. 

Data submitted by; 
Centers for Medicare & Medicaid Services (CMS): Hospitals paid under 
the Inpatient Prospective Payment System; 
Other reporting systems: American College of Cardiology (ACC): 
Facilities with at least one catheterization laboratory (includes in-
hospital, freestanding, and/or mobile catheterization laboratories); 
Other reporting systems: California Office of Statewide Health Planning 
and Development: Hospitals where cardiac surgeries are performed; 
Other reporting systems: Data Assessment and Verification Project 
(DAVE)[A]: Nursing homes; 
Other reporting systems: Joint Commission on Accreditation of 
Healthcare Organizations (JCAHO)[B]: JCAHO-accredited hospitals; 
Other reporting systems: National Committee for Quality Assurance 
(NCQA): Health plans; 
Other reporting systems: New York State Department of Health: Hospitals 
that perform cardiac surgery and/or percutaneous coronary intervention 
(PCI); 
Other reporting systems: Society of Thoracic Surgeons (STS): Hospitals, 
surgeons. 

Reporting requirement; 
Centers for Medicare & Medicaid Services (CMS): [C]; 
Other reporting systems: American College of Cardiology (ACC): 
Voluntary[D]; 
Other reporting systems: California Office of Statewide Health Planning 
and Development: Mandatory; 
Other reporting systems: Data Assessment and Verification Project 
(DAVE)[A]: Mandatory; 
Other reporting systems: Joint Commission on Accreditation of 
Healthcare Organizations (JCAHO)[B]: Mandatory[E]; 
Other reporting systems: National Committee for Quality Assurance 
(NCQA): Mandatory[E]; 
Other reporting systems: New York State Department of Health: 
Mandatory; 
Other reporting systems: Society of Thoracic Surgeons (STS): Voluntary. 

Are the data publicly reported? 
Centers for Medicare & Medicaid Services (CMS): Yes; 
Other reporting systems: American College of Cardiology (ACC): No; 
Other reporting systems: California Office of Statewide Health Planning 
and Development: Yes; 
Other reporting systems: Data Assessment and Verification Project 
(DAVE)[A]: Yes; 
Other reporting systems: Joint Commission on Accreditation of 
Healthcare Organizations (JCAHO)[B]: Yes; 
Other reporting systems: National Committee for Quality Assurance 
(NCQA): Yes[F]; 
Other reporting systems: New York State Department of Health: Yes; 
Other reporting systems: Society of Thoracic Surgeons (STS): No. 

Types of conditions for which data are submitted; 
Centers for Medicare & Medicaid Services (CMS): Cardiac-acute 
myocardial infarction (AMI), heart failure (HF); Pneumonia; 
Other reporting systems: American College of Cardiology (ACC): Cardiac-
diagnostic cardiac catheterization, PCI; 
Other reporting systems: California Office of Statewide Health Planning 
and Development: Cardiac-coronary artery bypass grafting (CABG); 
Other reporting systems: Data Assessment and Verification Project 
(DAVE)[A]: Resident health care; Resident health status; 
Other reporting systems: Joint Commission on Accreditation of 
Healthcare Organizations (JCAHO)[B]: Cardiac-AMI, HF; Pneumonia; 
Pregnancy; 
Surgical infection prevention; 
Other reporting systems: National Committee for Quality Assurance 
(NCQA): Preventive care, acute and chronic conditions; 
Other reporting systems: New York State Department of Health: Cardiac-
CABG, PCI, and valve surgery; 
Other reporting systems: Society of Thoracic Surgeons (STS): Cardiac-
CABG, aortic and mitral valve; General thoracic surgery; Congenital 
heart surgery. 

Number of facilities reporting; 
Centers for Medicare & Medicaid Services (CMS): 3,839[G]; 
Other reporting systems: American College of Cardiology (ACC): 611[H]; 
Other reporting systems: California Office of Statewide Health Planning 
and Development: 120; 
Other reporting systems: Data Assessment and Verification Project 
(DAVE)[A]: 16,266[I]; 
Other reporting systems: Joint Commission on Accreditation of 
Healthcare Organizations (JCAHO)[B]: ~3,350; 
Other reporting systems: National Committee for Quality Assurance 
(NCQA): 560; 
Other reporting systems: New York State Department of Health: 49; 
Other reporting systems: Society of Thoracic Surgeons (STS): 700. 

Approximate program duration; 
Centers for Medicare & Medicaid Services (CMS): 2 years; 
Other reporting systems: American College of Cardiology (ACC): 7 years; 
Other reporting systems: California Office of Statewide Health Planning 
and Development: 2 years[J]; 
Other reporting systems: Data Assessment and Verification Project 
(DAVE)[A]: 1 year; 
Other reporting systems: Joint Commission on Accreditation of 
Healthcare Organizations (JCAHO)[B]: 3 years; 
Other reporting systems: National Committee for Quality Assurance 
(NCQA): 14 years; 
Other reporting systems: New York State Department of Health: 16 years; 
Other reporting systems: Society of Thoracic Surgeons (STS): 16 years. 

Sources: CMS, ACC, California Office of Statewide Health Planning and 
Development, JCAHO, NCQA, New York State Department of Health, and STS. 

[A] DAVE is a CMS contract to assess the reliability of minimum data 
set assessment data that are submitted by nursing homes. Minimum data 
set assessments are a minimum data set of core elements to use in 
conducting comprehensive assessments of patient conditions and care 
needs. These assessments are collected for all residents in nursing 
homes that serve Medicare and Medicaid beneficiaries. 

[B] JCAHO provided information about its ORYX‚ initiative, which 
integrates outcome and other performance measurement data into the 
accreditation process. 

[C] Under Section 501(b) of the Medicare Prescription Drug, 
Improvement, and Modernization Act of 2003, hospitals shall submit data 
for a set of indicators established by the Department of Health and 
Human Services (HHS) as of November 1, 2003, related to the quality of 
inpatient care. Section 501 (b) also provides that any hospital that 
does not submit data on the 10 quality measures specified by the 
Secretary of Health and Human Services will have its annual payment 
update reduced by 0.4 percentage points for each fiscal year from 2005 
through 2007. 

[D] Some states and insurance companies have started to require 
hospital participation. 

[E] Data submission is mandatory to maintain accreditation. 

[F] Only audited data are publicly reported. 

[G] The number of hospitals that submitted data to receive their annual 
payment update for fiscal year 2005. 

[H] The number of facilities enrolled in ACC's National Cardiovascular 
Data Registry® as of July 13, 2005. 

[I] This number represents the number of nursing homes that submitted 
minimum data set assessments between January 1, 2004, and December 31, 
2004. Accuracy estimates are made by selecting a random sample of 
records for off-site and on-site medical record review. 

[J] Mandatory reporting of performance data began in 2003. 

[End of table] 

Table 4: Processes Used by CMS and Other Reporting Systems to Ensure 
Data Accuracy: 

[See PDF for image] 

Sources: CMS, ACC, California Office of Statewide Health Planning and 
Development, JCAHO, NCQA, New York State Department of Health, and STS. 

[A] DAVE is a CMS contract to assess the reliability of minimum data 
set assessment data that are submitted by nursing homes. Minimum data 
set assessments are a minimum data set of core elements to use in 
conducting comprehensive assessments of patient conditions and care 
needs. These assessments are collected for all residents in nursing 
homes that serve Medicare and Medicaid beneficiaries. 

[B] JCAHO provided information about its ORYX‚ initiative, which 
integrates outcome and other performance measurement data into the 
accreditation process. 

[C] CMS and JCAHO have worked to align their measures. A common set of 
measures took effect for discharges occurring on or after January 1, 
2005. 

[D] Data checks occur at the state level, for example, the state health 
department, before the data are accessed by DAVE. 

[E] JCAHO performs independent audits of data vendors. 

[F] STS is planning to incorporate an independent audit into its 
system. STS officials plan on including an on-side audit and medical 
record review as part of their audit system. 

[G] The 10 percent random sample of medical records is based on annual 
percutaneous coronary intervention volume. 

[H] The number of cases and facilities identified are limited to on-
site audits. Additional cases are reviewed as part of the off-site 
medical record review process. 

[I] Auditors review 100 percent of records when significant 
discrepancies are identified between the chart and what the hospital 
reported on specific risk factors. In addition, medical record 
documentation is reviewed for 100 percent of cases with the risk 
factors "shock" or "stent thrombosis". 

[J] STS plans to review a minimum of 30 records as a part of its 
independent auditing process. 

[K] ACC defines eligible sites as those facilities with a minimum of 50 
records to be abstracted over a specified number of quarters. 

[L] New York State Department of Health typically reviews 20 programs 
per year. In some instances that can mean percutaneous coronary 
intervention and cardiac surgery at the same hospital, which would 
count as two programs. 

[M] STS plans on visiting 24 facilities per year as a part of its 
independent auditing process. 

[End of table] 

Table 5: Processes Used by CMS and Other Reporting Systems to Ensure 
Data Completeness: 

[See PDF for image] 

Sources: CMS, ACC, California Office of Statewide Health Planning and 
Development, JCAHO, NCQA, New York State Department of Health, and STS. 

[A] DAVE is a CMS contract to assess the reliability of minimum data 
set assessment data that are submitted by nursing homes. Minimum data 
set assessments are a minimum data set of core elements to use in 
conducting comprehensive assessments of patient conditions and care 
needs. These assessments are collected for all residents in nursing 
homes that serve Medicare and Medicaid beneficiaries. 

[B] JCAHO provided information about its ORYX‚ initiative, which 
integrates outcome and other performance measurement data into the 
accreditation process. 

[C] Under concurrent review, auditors assess data as they are being 
collected. 

[D] JCAHO performs independent audits of data vendors. 

[E] STS is planning to incorporate an independent audit into its 
system. STS officials plan on including an on-side audit as part of 
their audit system. 

[F] The International Classification of Diseases, Ninth Revision (ICD-
9) codes were designed to promote international comparability in the 
collection, processing, classification, and presentation of mortality 
statistics. 

[G] CMS conducted two separate one-time studies that compared Medicare 
claims data to submitted data. 

[H] Data completeness reviews are conducted annually for randomly 
selected sites as part of the on-site audit process and quarterly for 
data submissions. 

[I] A one-time study was conducted; additional studies are planned. 

[J] At a minimum, data completeness reviews are conducted annually. 

[K] A one-time study was conducted. 

[End of table] 

[End of section] 

Appendix III: Data Tables on Hospital Accuracy Scores: 

Rural hospitals and smaller hospitals generally received accuracy 
scores that differed minimally from those of urban hospitals and larger 
hospitals. (See tables 6 and 7.) To the extent there are small 
differences across categories, they do not show a consistent pattern 
based on geographic location or size. 

Table 6: Median Hospital Baseline Accuracy Scores, by Hospital 
Characteristic, Quarter, and Measure Set: 

Hospital characteristic: Urban; 
January-March 2004 discharges: Median accuracy score for APU-measure 
set: 92.7; 
January-March 2004 discharges: Median accuracy score for expanded-
measure set: 90.0; 
April-June 2004 discharges: Median accuracy score for APU-measure set: 
94.2; 
April-June 2004 discharges: Median accuracy score for expanded-measure 
set: 91.5. 

Hospital characteristic: Rural; 
January-March 2004 discharges: Median accuracy score for APU-measure 
set: 93.0; 
January-March 2004 discharges: Median accuracy score for expanded-
measure set: 91.1; 
April-June 2004 discharges: Median accuracy score for APU-measure set: 
93.8; 
April-June 2004 discharges: Median accuracy score for expanded-measure 
set: 91.7. 

Hospital characteristic: < 50 beds; 
January-March 2004 discharges: Median accuracy score for APU-measure 
set: 93.0; 
January-March 2004 discharges: Median accuracy score for expanded-
measure set: 91.2; 
April-June 2004 discharges: Median accuracy score for APU-measure set: 
93.9; 
April-June 2004 discharges: Median accuracy score for expanded-measure 
set: 91.8. 

Hospital characteristic: 50-99 beds; 
January-March 2004 discharges: Median accuracy score for APU-measure 
set: 93.2; 
January-March 2004 discharges: Median accuracy score for expanded-
measure set: 91.1; 
April-June 2004 discharges: Median accuracy score for APU-measure set: 
94.2; 
April-June 2004 discharges: Median accuracy score for expanded-measure 
set: 92.2. 

Hospital characteristic: 100-199 beds; 
January-March 2004 discharges: Median accuracy score for APU-measure 
set: 92.9; 
January-March 2004 discharges: Median accuracy score for expanded-
measure set: 90.5; 
April-June 2004 discharges: Median accuracy score for APU-measure set: 
94.1; 
April-June 2004 discharges: Median accuracy score for expanded-measure 
set: 91.3. 

Hospital characteristic: 200-299 beds; 
January-March 2004 discharges: Median accuracy score for APU-measure 
set: 93.0; 
January-March 2004 discharges: Median accuracy score for expanded-
measure set: 90.1; 
April-June 2004 discharges: Median accuracy score for APU-measure set: 
94.2; 
April-June 2004 discharges: Median accuracy score for expanded-measure 
set: 91.7. 

Hospital characteristic: 300-399 beds; 
January-March 2004 discharges: Median accuracy score for APU-measure 
set: 92.7; 
January-March 2004 discharges: Median accuracy score for expanded-
measure set: 89.8; 
April-June 2004 discharges: Median accuracy score for APU-measure set: 
93.9; 
April-June 2004 discharges: Median accuracy score for expanded-measure 
set: 91.0. 

Hospital characteristic: 400-499 beds; 
January-March 2004 discharges: Median accuracy score for APU-measure 
set: 92.0; 
January-March 2004 discharges: Median accuracy score for expanded-
measure set: 89.5; 
April-June 2004 discharges: Median accuracy score for APU-measure set: 
93.8; 
April-June 2004 discharges: Median accuracy score for expanded-measure 
set: 91.1. 

Hospital characteristic: 500+ beds; 
January-March 2004 discharges: Median accuracy score for APU-measure 
set: 92.0; 
January-March 2004 discharges: Median accuracy score for expanded-
measure set: 89.0; 
April-June 2004 discharges: Median accuracy score for APU-measure set: 
94.1; 
April-June 2004 discharges: Median accuracy score for expanded-measure 
set: 91.0. 

Hospital characteristic: All hospitals; 
January-March 2004 discharges: Median accuracy score for APU-measure 
set: 92.9; 
January-March 2004 discharges: Median accuracy score for expanded-
measure set: 90.4; 
April-June 2004 discharges: Median accuracy score for APU-measure set: 
94.1; 
April-June 2004 discharges: Median accuracy score for expanded-measure 
set: 91.6. 

Source: GAO analysis of CMS data. 

Note: Calculation of accuracy scores for the expanded-measure set was 
based on all the measures for which a hospital submitted data, which 
could range from the APU measures alone to a maximum of 17--the APU 
measures plus as many as 7 additional measures. 

[End of table] 

Table 7: Proportion of Hospitals with Baseline Accuracy Scores Not 
Meeting 80 Percent Threshold, by Hospital Characteristic, Quarter, and 
Measure Set: 

Hospital characteristic: Urban; 
January-March 2004 discharges: Percentage not meeting threshold for APU-
measure set: 10.3; 
January-March 2004 discharges: Percentage not meeting threshold for 
expanded-measure set: 14.4; 
April-June 2004 discharges: Percentage not meeting threshold for APU-
measure set: 7.7; 
April-June 2004 discharges: Percentage not meeting threshold for 
expanded-measure set: 10.3. 

Hospital characteristic: Rural; 
January-March 2004 discharges: Percentage not meeting threshold for APU-
measure set: 9.1; 
January-March 2004 discharges: Percentage not meeting threshold for 
expanded-measure set: 11.6; 
April-June 2004 discharges: Percentage not meeting threshold for APU-
measure set: 8.9; 
April-June 2004 discharges: Percentage not meeting threshold for 
expanded-measure set: 9.6. 

Hospital characteristic: < 50 beds; 
January-March 2004 discharges: Percentage not meeting threshold for APU-
measure set: 9.4; 
January-March 2004 discharges: Percentage not meeting threshold for 
expanded-measure set: 12.8; 
April-June 2004 discharges: Percentage not meeting threshold for APU-
measure set: 10.3; 
April-June 2004 discharges: Percentage not meeting threshold for 
expanded-measure set: 12.0. 

Hospital characteristic: 50-99 beds; 
January-March 2004 discharges: Percentage not meeting threshold for APU-
measure set: 9.6; 
January-March 2004 discharges: Percentage not meeting threshold for 
expanded-measure set: 12.4; 
April-June 2004 discharges: Percentage not meeting threshold for APU-
measure set: 8.3; 
April-June 2004 discharges: Percentage not meeting threshold for 
expanded-measure set: 8.5. 

Hospital characteristic: 100-199 beds; 
January-March 2004 discharges: Percentage not meeting threshold for APU-
measure set: 8.7; 
January-March 2004 discharges: Percentage not meeting threshold for 
expanded-measure set: 12.3; 
April-June 2004 discharges: Percentage not meeting threshold for APU-
measure set: 8.6; 
April-June 2004 discharges: Percentage not meeting threshold for 
expanded-measure set: 9.8. 

Hospital characteristic: 200-299 beds; 
January-March 2004 discharges: Percentage not meeting threshold for APU-
measure set: 9.5; 
January-March 2004 discharges: Percentage not meeting threshold for 
expanded-measure set: 12.8; 
April-June 2004 discharges: Percentage not meeting threshold for APU-
measure set: 6.0; 
April-June 2004 discharges: Percentage not meeting threshold for 
expanded-measure set: 9.3. 

Hospital characteristic: 300-399 beds; 
January-March 2004 discharges: Percentage not meeting threshold for APU-
measure set: 11.8; 
January-March 2004 discharges: Percentage not meeting threshold for 
expanded-measure set: 15.0; 
April-June 2004 discharges: Percentage not meeting threshold for APU-
measure set: 6.5; 
April-June 2004 discharges: Percentage not meeting threshold for 
expanded-measure set: 8.6. 

Hospital characteristic: 400-499 beds; 
January-March 2004 discharges: Percentage not meeting threshold for APU-
measure set: 10.6; 
January-March 2004 discharges: Percentage not meeting threshold for 
expanded-measure set: 14.1; 
April-June 2004 discharges: Percentage not meeting threshold for APU-
measure set: 8.1; 
April-June 2004 discharges: Percentage not meeting threshold for 
expanded-measure set: 11.1. 

Hospital characteristic: 500+ beds; 
January-March 2004 discharges: Percentage not meeting threshold for APU-
measure set: 12.2; 
January-March 2004 discharges: Percentage not meeting threshold for 
expanded-measure set: 16.6; 
April-June 2004 discharges: Percentage not meeting threshold for APU-
measure set: 8.6; 
April-June 2004 discharges: Percentage not meeting threshold for 
expanded-measure set: 12.2. 

Hospital characteristic: All hospitals; 
January-March 2004 discharges: Percentage not meeting threshold for APU-
measure set: 9.8; 
January-March 2004 discharges: Percentage not meeting threshold for 
expanded-measure set: 13.2; 
April-June 2004 discharges: Percentage not meeting threshold for APU-
measure set: 8.2; 
April-June 2004 discharges: Percentage not meeting threshold for 
expanded-measure set: 10.0. 

Source: GAO analysis of CMS data. 

Note: Calculation of accuracy scores for the expanded-measure set was 
based on all the measures for which a hospital submitted data, which 
could range from the APU measures alone to a maximum of 17--the APU 
measures plus as many as 7 additional measures. CMS deems hospitals 
that achieve an accuracy score of 80 or better as having met its 
requirement to submit accurate data. 

[End of table] 

Accuracy scores among hospitals whose data were submitted to CMS by 
different JCAHO-certified vendors varied more, especially in the 
percentage of the hospitals that failed to meet the 80 percent 
threshold. (See table 8.) Collectively, these data vendors submitted 
data to the clinical warehouse for approximately 78 to 79 percent of 
hospitals affected by the APU program in the two baseline quarters, 
while another 13 to 14 percent of hospitals directly submitted their 
own data. For large data vendors (serving more than 100 hospitals), 
medium vendors (serving between 20 and 100 hospitals), and small 
vendors (serving fewer than 20 hospitals), there was marked variation 
within each size grouping in the proportion of the vendors' hospitals 
that did not meet the accuracy threshold. Such variation could reflect 
differences in the hospitals served by different vendors as well as 
differences in the services provided by those vendors. 

Table 8: Percentage of Hospitals with Baseline Accuracy Scores Not 
Meeting 80 Percent Threshold, by JCAHO-Certified Vendor Grouped by 
Number of Hospitals Served, Quarter, and Measure Set: 

Large vendors: 

Vendors, grouped by number of hospitals served: Vendor 1; 
Percentage not meeting threshold for APU-measure set: January-March 
2004 discharges: 2.6; 
Percentage not meeting threshold for APU-measure set: April-June 2004 
discharges: 2.6; 
Percentage not meeting threshold for expanded-measure set: January-
March 2004 discharges: 3.9; 
Percentage not meeting threshold for expanded-measure set: April-June 
2004 discharges: 2.6. 

Vendors, grouped by number of hospitals served: Vendor 2; 
Percentage not meeting threshold for APU-measure set: January-March 
2004 discharges: 7.1; 
Percentage not meeting threshold for APU-measure set: April-June 2004 
discharges: 7.2; 
Percentage not meeting threshold for expanded-measure set: January-
March 2004 discharges: 9.3; 
Percentage not meeting threshold for expanded-measure set: April-June 
2004 discharges: 7.2. 

Vendors, grouped by number of hospitals served: Vendor 3; 
Percentage not meeting threshold for APU-measure set: January-March 
2004 discharges: 7.7; 
Percentage not meeting threshold for APU-measure set: April-June 2004 
discharges: 9.5; 
Percentage not meeting threshold for expanded-measure set: January-
March 2004 discharges: 14.0; 
Percentage not meeting threshold for expanded-measure set: April-June 
2004 discharges: 11.3. 

Vendors, grouped by number of hospitals served: Vendor 4; 
Percentage not meeting threshold for APU-measure set: January-March 
2004 discharges: 10.1; 
Percentage not meeting threshold for APU-measure set: April-June 2004 
discharges: 9.8; 
Percentage not meeting threshold for expanded-measure set: January-
March 2004 discharges: 11.1; 
Percentage not meeting threshold for expanded-measure set: April-June 
2004 discharges: 10.2. 

Vendors, grouped by number of hospitals served: Vendor 5; 
Percentage not meeting threshold for APU-measure set: January-March 
2004 discharges: 11.1; 
Percentage not meeting threshold for APU-measure set: April-June 2004 
discharges: 8.4; 
Percentage not meeting threshold for expanded-measure set: January-
March 2004 discharges: 14.4; 
Percentage not meeting threshold for expanded-measure set: April-June 
2004 discharges: 10.4. 

Vendors, grouped by number of hospitals served: Vendor 6; 
Percentage not meeting threshold for APU-measure set: January-March 
2004 discharges: 12.2; 
Percentage not meeting threshold for APU-measure set: April-June 2004 
discharges: 10.4; 
Percentage not meeting threshold for expanded-measure set: January-
March 2004 discharges: 16.5; 
Percentage not meeting threshold for expanded-measure set: April-June 
2004 discharges: 11.3. 

Vendors, grouped by number of hospitals served: Vendor 7; 
Percentage not meeting threshold for APU-measure set: January-March 
2004 discharges: 12.4; 
Percentage not meeting threshold for APU-measure set: April-June 2004 
discharges: 9.0; 
Percentage not meeting threshold for expanded-measure set: January-
March 2004 discharges: 12.4; 
Percentage not meeting threshold for expanded-measure set: April-June 
2004 discharges: 13.6. 

Vendors, grouped by number of hospitals served: Vendor 8; 
Percentage not meeting threshold for APU-measure set: January-March 
2004 discharges: 13.3; 
Percentage not meeting threshold for APU-measure set: April-June 2004 
discharges: 5.8; 
Percentage not meeting threshold for expanded-measure set: January-
March 2004 discharges: 15.8; 
Percentage not meeting threshold for expanded-measure set: April-June 
2004 discharges: 7.9. 

Medium vendors: 

Vendors, grouped by number of hospitals served: Vendor 9; 
Percentage not meeting threshold for APU-measure set: January-March 
2004 discharges: 2.4; 
Percentage not meeting threshold for APU-measure set: April-June 2004 
discharges: 4.5; 
Percentage not meeting threshold for expanded-measure set: January-
March 2004 discharges: 2.4; 
Percentage not meeting threshold for expanded-measure set: April-June 
2004 discharges: 2.3. 

Vendors, grouped by number of hospitals served: Vendor 10; 
Percentage not meeting threshold for APU-measure set: January-March 
2004 discharges: 3.4; 
Percentage not meeting threshold for APU-measure set: April-June 2004 
discharges: 3.1; 
Percentage not meeting threshold for expanded-measure set: January-
March 2004 discharges: 3.4; 
Percentage not meeting threshold for expanded-measure set: April-June 
2004 discharges: 6.3. 

Vendors, grouped by number of hospitals served: Vendor 11; 
Percentage not meeting threshold for APU-measure set: January-March 
2004 discharges: 4.2; 
Percentage not meeting threshold for APU-measure set: April-June 2004 
discharges: 6.8; 
Percentage not meeting threshold for expanded-measure set: January-
March 2004 discharges: 6.9; 
Percentage not meeting threshold for expanded-measure set: April-June 
2004 discharges: 6.8. 

Vendors, grouped by number of hospitals served: Vendor 12; 
Percentage not meeting threshold for APU-measure set: January-March 
2004 discharges: 4.8; 
Percentage not meeting threshold for APU-measure set: April-June 2004 
discharges: 4.8; 
Percentage not meeting threshold for expanded-measure set: January-
March 2004 discharges: 4.8; 
Percentage not meeting threshold for expanded-measure set: April-June 
2004 discharges: 6.5. 

Vendors, grouped by number of hospitals served: Vendor 13; 
Percentage not meeting threshold for APU-measure set: January-March 
2004 discharges: 4.9; 
Percentage not meeting threshold for APU-measure set: April-June 2004 
discharges: 2.8; 
Percentage not meeting threshold for expanded-measure set: January-
March 2004 discharges: 4.9; 
Percentage not meeting threshold for expanded-measure set: April-June 
2004 discharges: 2.8. 

Vendors, grouped by number of hospitals served: Vendor 14; 
Percentage not meeting threshold for APU-measure set: January-March 
2004 discharges: 6.4; 
Percentage not meeting threshold for APU-measure set: April-June 2004 
discharges: 4.3; 
Percentage not meeting threshold for expanded-measure set: January-
March 2004 discharges: 8.5; 
Percentage not meeting threshold for expanded-measure set: April-June 
2004 discharges: 6.4. 

Vendors, grouped by number of hospitals served: Vendor 15; 
Percentage not meeting threshold for APU-measure set: January-March 
2004 discharges: 7.1; 
Percentage not meeting threshold for APU-measure set: April-June 2004 
discharges: 6.0; 
Percentage not meeting threshold for expanded-measure set: January-
March 2004 discharges: 7.1; 
Percentage not meeting threshold for expanded-measure set: April-June 
2004 discharges: 7.5. 

Vendors, grouped by number of hospitals served: Vendor 16; 
Percentage not meeting threshold for APU-measure set: January-March 
2004 discharges: 7.6; 
Percentage not meeting threshold for APU-measure set: April-June 2004 
discharges: 5.0; 
Percentage not meeting threshold for expanded-measure set: January-
March 2004 discharges: 19.0; 
Percentage not meeting threshold for expanded-measure set: April-June 
2004 discharges: 13.8. 

Vendors, grouped by number of hospitals served: Vendor 17; 
Percentage not meeting threshold for APU-measure set: January-March 
2004 discharges: 7.9; 
Percentage not meeting threshold for APU-measure set: April-June 2004 
discharges: 2.6; 
Percentage not meeting threshold for expanded-measure set: January-
March 2004 discharges: 9.2; 
Percentage not meeting threshold for expanded-measure set: April-June 
2004 discharges: 2.6. 

Vendors, grouped by number of hospitals served: Vendor 18; 
Percentage not meeting threshold for APU-measure set: January-March 
2004 discharges: 8.0; 
Percentage not meeting threshold for APU-measure set: April-June 2004 
discharges: 3.4; 
Percentage not meeting threshold for expanded-measure set: January-
March 2004 discharges: 12.0; 
Percentage not meeting threshold for expanded-measure set: April-June 
2004 discharges: 6.9. 

Vendors, grouped by number of hospitals served: Vendor 19; 
Percentage not meeting threshold for APU-measure set: January-March 
2004 discharges: 8.8; 
Percentage not meeting threshold for APU-measure set: April-June 2004 
discharges: 2.9; 
Percentage not meeting threshold for expanded-measure set: January-
March 2004 discharges: 26.5; 
Percentage not meeting threshold for expanded-measure set: April-June 
2004 discharges: 8.8. 

Vendors, grouped by number of hospitals served: Vendor 20; 
Percentage not meeting threshold for APU-measure set: January-March 
2004 discharges: 12.1; 
Percentage not meeting threshold for APU-measure set: April-June 2004 
discharges: 5.5; 
Percentage not meeting threshold for expanded-measure set: January-
March 2004 discharges: 17.6; 
Percentage not meeting threshold for expanded-measure set: April-June 
2004 discharges: 7.7. 

Vendors, grouped by number of hospitals served: Vendor 21; 
Percentage not meeting threshold for APU-measure set: January-March 
2004 discharges: 13.5; 
Percentage not meeting threshold for APU-measure set: April-June 2004 
discharges: 5.6; 
Percentage not meeting threshold for expanded-measure set: January-
March 2004 discharges: 13.5; 
Percentage not meeting threshold for expanded-measure set: April-June 
2004 discharges: 8.3. 

Vendors, grouped by number of hospitals served: Vendor 22; 
Percentage not meeting threshold for APU-measure set: January-March 
2004 discharges: 15.2; 
Percentage not meeting threshold for APU-measure set: April-June 2004 
discharges: 13.9; 
Percentage not meeting threshold for expanded-measure set: January-
March 2004 discharges: 17.7; 
Percentage not meeting threshold for expanded-measure set: April-June 
2004 discharges: 17.7. 

Vendors, grouped by number of hospitals served: Vendor 23; 
Percentage not meeting threshold for APU-measure set: January-March 
2004 discharges: 18.4; 
Percentage not meeting threshold for APU-measure set: April-June 2004 
discharges: 10.0; 
Percentage not meeting threshold for expanded-measure set: January-
March 2004 discharges: 28.6; 
Percentage not meeting threshold for expanded-measure set: April-June 
2004 discharges: 12.0. 

Small vendors: 

Vendors, grouped by number of hospitals served: Vendor 24; 
Percentage not meeting threshold for APU-measure set: January-March 
2004 discharges: 0.0; 
Percentage not meeting threshold for APU-measure set: April-June 2004 
discharges: 11.8; 
Percentage not meeting threshold for expanded-measure set: January-
March 2004 discharges: 0.0; 
Percentage not meeting threshold for expanded-measure set: April-June 
2004 discharges: 11.8. 

Vendors, grouped by number of hospitals served: Vendor 25; 
Percentage not meeting threshold for APU-measure set: January-March 
2004 discharges: 0.0; 
Percentage not meeting threshold for APU-measure set: April-June 2004 
discharges: 7.1; 
Percentage not meeting threshold for expanded-measure set: January-
March 2004 discharges: 0.0; 
Percentage not meeting threshold for expanded-measure set: April-June 
2004 discharges: 7.1. 

Vendors, grouped by number of hospitals served: Vendor 26; 
Percentage not meeting threshold for APU-measure set: January-March 
2004 discharges: 0.0; 
Percentage not meeting threshold for APU-measure set: April-June 2004 
discharges: 0.0; 
Percentage not meeting threshold for expanded-measure set: January-
March 2004 discharges: 0.0; 
Percentage not meeting threshold for expanded-measure set: April-June 
2004 discharges: 0.0. 

Vendors, grouped by number of hospitals served: Vendor 27; 
Percentage not meeting threshold for APU-measure set: January-March 
2004 discharges: 0.0; 
Percentage not meeting threshold for APU-measure set: April-June 2004 
discharges: 16.7; 
Percentage not meeting threshold for expanded-measure set: January-
March 2004 discharges: 0.0; 
Percentage not meeting threshold for expanded-measure set: April-June 
2004 discharges: 16.7. 

Vendors, grouped by number of hospitals served: Vendor 28; 
Percentage not meeting threshold for APU-measure set: January-March 
2004 discharges: 0.0; 
Percentage not meeting threshold for APU-measure set: April-June 2004 
discharges: 0.0; 
Percentage not meeting threshold for expanded-measure set: January-
March 2004 discharges: 0.0; 
Percentage not meeting threshold for expanded-measure set: April-June 
2004 discharges: 0.0. 

Vendors, grouped by number of hospitals served: Vendor 29; 
Percentage not meeting threshold for APU-measure set: January-March 
2004 discharges: 0.0; 
Percentage not meeting threshold for APU-measure set: April-June 2004 
discharges: 0.0; 
Percentage not meeting threshold for expanded-measure set: January-
March 2004 discharges: 0.0; 
Percentage not meeting threshold for expanded-measure set: April-June 
2004 discharges: 0.0. 

Vendors, grouped by number of hospitals served: Vendor 30; 
Percentage not meeting threshold for APU-measure set: January-March 
2004 discharges: 0.0; 
Percentage not meeting threshold for APU-measure set: April-June 2004 
discharges: 0.0; 
Percentage not meeting threshold for expanded-measure set: January-
March 2004 discharges: 0.0; 
Percentage not meeting threshold for expanded-measure set: April-June 
2004 discharges: 0.0. 

Vendors, grouped by number of hospitals served: Vendor 31; 
Percentage not meeting threshold for APU-measure set: January-March 
2004 discharges: 8.3; 
Percentage not meeting threshold for APU-measure set: April-June 2004 
discharges: 0.0; 
Percentage not meeting threshold for expanded-measure set: January-
March 2004 discharges: 16.7; 
Percentage not meeting threshold for expanded-measure set: April-June 
2004 discharges: 0.0. 

Vendors, grouped by number of hospitals served: Vendor 32; 
Percentage not meeting threshold for APU-measure set: January-March 
2004 discharges: 9.1; 
Percentage not meeting threshold for APU-measure set: April-June 2004 
discharges: 8.3; 
Percentage not meeting threshold for expanded-measure set: January-
March 2004 discharges: 9.1; 
Percentage not meeting threshold for expanded-measure set: April-June 
2004 discharges: 16.7. 

Vendors, grouped by number of hospitals served: Vendor 33; 
Percentage not meeting threshold for APU-measure set: January-March 
2004 discharges: 9.1; 
Percentage not meeting threshold for APU-measure set: April-June 2004 
discharges: 0.0; 
Percentage not meeting threshold for expanded-measure set: January-
March 2004 discharges: 27.3; 
Percentage not meeting threshold for expanded-measure set: April-June 
2004 discharges: 0.0. 

Vendors, grouped by number of hospitals served: Vendor 34; 
Percentage not meeting threshold for APU-measure set: January-March 
2004 discharges: 10.0; 
Percentage not meeting threshold for APU-measure set: April-June 2004 
discharges: 9.1; 
Percentage not meeting threshold for expanded-measure set: January-
March 2004 discharges: 10.0; 
Percentage not meeting threshold for expanded-measure set: April-June 
2004 discharges: 9.1. 

Vendors, grouped by number of hospitals served: Vendor 35; 
Percentage not meeting threshold for APU-measure set: January-March 
2004 discharges: 11.1; 
Percentage not meeting threshold for APU-measure set: April-June 2004 
discharges: 11.1; 
Percentage not meeting threshold for expanded-measure set: January-
March 2004 discharges: 11.1; 
Percentage not meeting threshold for expanded-measure set: April-June 
2004 discharges: 11.1. 

Vendors, grouped by number of hospitals served: Vendor 36; 
Percentage not meeting threshold for APU-measure set: January-March 
2004 discharges: 20.0; 
Percentage not meeting threshold for APU-measure set: April-June 2004 
discharges: 33.3; 
Percentage not meeting threshold for expanded-measure set: January-
March 2004 discharges: 60.0; 
Percentage not meeting threshold for expanded-measure set: April-June 
2004 discharges: 33.3. 

Vendors, grouped by number of hospitals served: Vendor 37; 
Percentage not meeting threshold for APU-measure set: January-March 
2004 discharges: 33.3; 
Percentage not meeting threshold for APU-measure set: April-June 2004 
discharges: 0.0; 
Percentage not meeting threshold for expanded-measure set: January-
March 2004 discharges: 33.3; 
Percentage not meeting threshold for expanded-measure set: April-June 
2004 discharges: 0.0. 

Vendors, grouped by number of hospitals served: Vendor 38; 
Percentage not meeting threshold for APU-measure set: January-March 
2004 discharges: 33.3; 
Percentage not meeting threshold for APU-measure set: April-June 2004 
discharges: 0.0; 
Percentage not meeting threshold for expanded-measure set: January-
March 2004 discharges: 33.3; 
Percentage not meeting threshold for expanded-measure set: April-June 
2004 discharges: 0.0. 

Vendors, grouped by number of hospitals served: No vendor; 
Percentage not meeting threshold for APU-measure set: January-March 
2004 discharges: 10.2; 
Percentage not meeting threshold for APU-measure set: April-June 2004 
discharges: 12.5; 
Percentage not meeting threshold for expanded-measure set: January-
March 2004 discharges: 11.6; 
Percentage not meeting threshold for expanded-measure set: April-June 
2004 discharges: 13.2. 

Source: GAO analysis of CMS data. 

Note: Large vendors served more than 100 hospitals, medium vendors 
served 20 to 100 hospitals, and small vendors served fewer than 20 
hospitals. Calculation of accuracy scores for the expanded-measure set 
was based on all the measures for which a hospital submitted data, 
which could range from the APU measures alone to a maximum of 17--the 
APU measures plus as many as 7 additional measures. CMS deems hospitals 
that achieve an accuracy score of 80 or better as having met its 
requirement to submit accurate data. 

[End of table] 

Rank ordering hospitals by the breadth of the confidence intervals 
around their accuracy scores, from the narrowest to the widest 
intervals, shows the large variation that we found across both quarters 
and measure sets. Hospitals with the narrowest confidence intervals, 
shown in table 9 as the 10th percentile, had a range of no more than 6 
percentage points between the lower and upper limits of their 
confidence interval. That meant that their accuracy scores from one 
sample to the next were likely to vary by no more than plus or minus 3 
percentage points from the accuracy score obtained in the sample drawn 
by CMS. By contrast, hospitals with the widest confidence intervals, 
shown in table 9 as the 90th percentile, exceeded 36 percentage points 
from the lower limit to the upper limit of their confidence interval. 
The accuracy scores for these hospitals would likely vary from one 
sample to the next by 18 percentage points or more, up or down, 
relative to the accuracy score derived from the CMS sample. For 
hospitals whose confidence interval included the 80 percent threshold, 
it was statistically uncertain whether a different sample of cases 
would have altered their result from passing the 80 percent threshold 
to failing, or vice versa. 

Table 9: Breadth of Confidence Intervals in Percentage Points Around 
the Hospital Baseline Accuracy Scores at Selected Percentiles, by 
Measure Set and Quarter: 

Hospital percentiles from narrowest to widest confidence intervals: 
10th percentile; 
APU-measure set: January-March 2004 discharges: 5.4; 
APU-measure set: April-June 2004 discharges: 0.0; 
Expanded-measure set: January-March 2004 discharges: 6.0; 
Expanded-measure set: April-June 2004 discharges: 5.6. 

Hospital percentiles from narrowest to widest confidence intervals: 
25th percentile; 
APU-measure set: January-March 2004 discharges: 8.1; 
APU-measure set: April-June 2004 discharges: 7.3; 
Expanded-measure set: January-March 2004 discharges: 9.3; 
Expanded-measure set: April-June 2004 discharges: 8.2. 

Hospital percentiles from narrowest to widest confidence intervals: 
Median; 
APU-measure set: January-March 2004 discharges: 14.0; 
APU-measure set: April-June 2004 discharges: 11.8; 
Expanded-measure set: January-March 2004 discharges: 14.6; 
Expanded-measure set: April-June 2004 discharges: 13.0. 

Hospital percentiles from narrowest to widest confidence intervals: 
75th percentile; 
APU-measure set: January-March 2004 discharges: 24.2; 
APU-measure set: April-June 2004 discharges: 21.5; 
Expanded-measure set: January-March 2004 discharges: 23.6; 
Expanded-measure set: April-June 2004 discharges: 21.3. 

Hospital percentiles from narrowest to widest confidence intervals: 
90th percentile; 
APU-measure set: January-March 2004 discharges: 40.3; 
APU-measure set: April-June 2004 discharges: 41.0; 
Expanded-measure set: January-March 2004 discharges: 37.9; 
Expanded-measure set: April-June 2004 discharges: 36.8. 

Source: GAO analysis of CMS data. 

Note: Confidence interval based on a 95 percent significance level. 
Calculation of accuracy scores and confidence intervals for the 
expanded-measure set was based on all the measures for which a hospital 
submitted data, which could range from the APU measures alone to a 
maximum of 17--the APU measures plus as many as 7 additional measures. 

[End of table] 

One-third to one-fourth of hospitals had statistically uncertain 
results because their confidence interval extended both above and below 
the 80 percent threshold. Some of these hospitals had accuracy scores 
of 80 or above and some had scores of less than 80. Table 10 separates 
these hospitals into (1) those that had accuracy scores equal to 80 or 
above and were statistically uncertain and (2) those that had accuracy 
scores below 80 and were statistically uncertain. The table shows that 
most of the statistical uncertainty involved hospitals that passed 
CMS's accuracy threshold, but if a different sample of cases had been 
reabstracted by CDAC, there was a substantial possibility that they 
would not have passed. 

Table 10: For Hospitals with Confidence Intervals That Included the 80 
Percent Threshold, Percentage of Total Hospitals with an Actual 
Baseline Accuracy Score That Either Met or Failed to Meet the 
Threshold, by Measure Set and Quarter: 

Percentage of hospitals whose actual accuracy score equals 80 or 
better; 
APU-measure set: January-March 2004 discharges: 23.9; 
APU-measure set: April-June 2004 discharges: 19.2; 
Expanded-measure set: January-March 2004 discharges: 28.0; 
Expanded-measure set: April-June 2004 discharges: 24.0. 

Percentage of hospitals whose actual accuracy score equals less than 
80; 
APU-measure set: January-March 2004 discharges: 8.3; 
APU-measure set: April-June 2004 discharges: 7.0; 
Expanded-measure set: January-March 2004 discharges: 11.3; 
Expanded-measure set: April-June 2004 discharges: 8.7. 

Total; 
APU-measure set: January-March 2004 discharges: 32.2; 
APU-measure set: April-June 2004 discharges: 26.3; 
Expanded-measure set: January-March 2004 discharges: 39.2; 
Expanded-measure set: April-June 2004 discharges: 32.7. 

Source: GAO analysis of CMS data. 

Note: Confidence interval based on a 95 percent significance level. 
Calculation of accuracy scores for the expanded-measure set was based 
on all the measures for which a hospital submitted data, which could 
range from the APU measures alone to a maximum of 17--the APU measures 
plus as many as 7 additional measures. CMS deems hospitals that achieve 
an accuracy score of 80 or better as having met its requirement to 
submit accurate data. 

[End of table] 

[End of section] 

Appendix IV: Comments from the Centers for Medicare & Medicaid 
Services: 

DEPARTMENT OF HEALTH & HUMAN SERVICES: 
Centers for Medicare & Medicaid Services: 
Administrator: 
Washington, DC 20201: 

DATE: DEC 9 2005: 

TO: Cynthia A. Bascetta: 
Director, Health Care: 
Government Accountability Office: 

Signed by: 

FROM: Mark B. McClellan, M.D., Ph.D.: 
Administrator: 
Centers for Medicare & Medicaid Services: 

SUBJECT: Government Accountability Office's (GAO) Draft Report: 
HOSPITAL QUALITY DATA: CMS Needs More Rigorous Methods to Ensure 
Reliability of Publicly Released Data (GAO-06-54): 

Thank you for the opportunity to review and comment on the above-
referenced subject report. The Centers for Medicare & Medicaid Services 
(CMS) welcomes GAO's finding of a high overall baseline accuracy rate 
when it examined CMS' assessment of the hospital quality data submitted 
for the Annual Payment Update (APU) program. With respect to GAO's 
finding that CMS has established no ongoing process to check data 
completeness, we do have a process in place. CMS checked data 
completeness annually during the 2 years of the APU program by 
assessing counts of hospital submitted data relative to Medicare claims 
submissions for all hospitals included in the program. 

The CMS instituted voluntary hospital reporting of quality data in 
2003, for the first time, as part of the Quality Improvement 
Organization (QIO) 7th Statement of Work. CMS required the QIOs to 
assist all hospitals nationwide with voluntary reporting of a set of 
quality measures to a clinical warehouse, and to help hospitals improve 
performance on these measures. The quality measures covered four 
clinical topics; Acute Myocardial Infarction, Heart Failure, Pneumonia, 
and Surgical Infection Prevention. 

Section 501 (b) of the Medicare Prescription Drug, Improvement, and 
Modernization Act of 2003 dramatically changed the environment for 
hospital reporting. Per this provision, Prospective Payment System 
(PPS) hospitals not submitting a set of 10 quality measures receive a 
reduced APU by 0.4 percent for Fiscal Years 2005, 2006, and 2007. 

The CMS successfully implemented and administered the APU program 
during 2004. The number of hospitals submitting data to the clinical 
warehouse increased dramatically, with nearly 99 percent of eligible 
hospitals participating. As a result of the APU, CMS modified the 
reporting program by adding a validation component under which CMS 
contractors assessed the accuracy of hospital chart abstraction for 
hospitals submitting data. Additionally. CMS assessed relative volume 
of hospital reporting by comparing reporting volume by hospitals to 
their Medicare claims submissions. As a result of this successful 
implementation, public reporting of hospital quality data for about 
3,600 hospitals on the Department of Health and Human Services Compare 
Web site was launched in 2004. Over 99 percent of PPS hospitals 
received the full APU payment in the 2004 detennination. 

In 2005, CMS successfully strengthened the accuracy criteria for the 
APU program. CMS expanded the APU criteria during 2005 to require 
hospitals with at least 6 eligible patients per quarter in the covered 
topics to submit accurate data. Through contractors, the CMS assessed 
the accuracy of submitted data, and determined that approximately 99 
percent of submitting hospitals achieved an 80 percent upper bound of 
confidence interval accuracy threshold. The CMS used only 2 quarters of 
data (3rd and 4th quarter 2004 discharges) because previous quarters' 
measures definitions were not completely aligned with the Joint 
Commission on Accreditation of Healthcare Organizations (JCAHO) 
measures. Prior to this reporting period, these measure definitional 
differences slightly impacted some JCAHO hospitals' accuracy results. 

Additionally in 2005, the CMS expanded its review of the completeness 
of data submissions by comparing the volume of submissions to claims 
data by quarter and by topic. CMS found that discrepancies between the 
volume of claims and data submissions in about 20 percent of hospitals. 
However, 96 percent of PPS hospitals submitted data for topics for 
which they had eligible cases for each quarter, and therefore received 
the full update in the 2005 determination. 

The CMS appreciates the thoughtful analysis and recommendations in the 
GAO report. The Agency believes that its methods to evaluate accuracy 
of submissions are sound, and agree that the quality and completeness 
of the data must be improved. This can best be accomplished through 
quarterly reports to hospitals that promote improvement in data 
accuracy. The Agency is also considering various other steps, as 
indicated in the detailed comments to the recommendations. 

Attached are the detailed comments to the GAO's recommendations. 

Centers for Medicare & Medicaid Services' (CMS) Comments to the 
Government Accountability Office's (GAO) Draft Report: HOSPITAL 
QUALITYDATA: CMS Needs More Rigorous Methods to Ensure Reliability of 
Publicly Released Data (GAO-06-54): 

In order for CMS to help ensure the reliability of the quality data it 
uses to produce information on hospital performance, GAO wrote three 
recommendations to CMS. Our responses follow each recommendation below: 

GAO Recommendation: 

Focusing on the subset of hospitals for which it is statistically 
uncertain if they met CMS accuracy threshold in one or more previous 
quarters, increase the number of patient records reabstracted by the 
clinical data abstraction center so that the proportion of hospitals 
with statistically uncertain results is reduced. 

CMS Response: 

The CMS believes that our methods to evaluate accuracy of submissions 
are sound. Our quarterly reabstraction sample of hospital submitted 
data found that hospital submitted data were generally accurate, as 
evident by approximately 90 percent of submitting hospitals achieving 
the 80 percent accuracy threshold. We agree that the quarterly accuracy 
estimates using 5 sample charts can have considerable sampling error, 
and is highly dependent on the clustering of errors within individual 
charts. As GAO cited in the report, confidence intervals generally 
range from 10 percent to 14 percent, but can be as high as 35 percent 
to 40 percent when errors are clustered in a single chart. 

Measure definition differences with the Joint Commission on 
Accreditation of Healthcare Organizations (JCAHO) prevented CMS from 
using accuracy results prior to 3rd quarter 2004 for the 2005 Annual 
Payment Update (APU) determination, since slight definitional 
differences might differentially impact accuracy rates. These 
definitional differences were resolved with the July 2005 and January 
2006 alignment modifications, where all hospitals submit data for the 
same measures. 

The quarterly accuracy estimates are primarily designed to provide 
hospitals and their vendors periodic feedback about the relative 
accuracy of their abstraction processing for quality improvement. We 
recommend that hospitals and vendors analyze accuracy results from 
multiple quarters to assess their abstraction processes. We will 
continue to educate hospitals and vendors to analyze all quarters' 
reabstraction accuracy results in order to provide a more reliable 
snapshot of their abstraction accuracy over time. 

Timing issues prevent CMS from implementing a targeted reabstraction 
subsampling of hospitals falling into the statistically uncertain 
outcome range. Identifying these hospitals requires completion of the 
initial 90 day reabstraction and appeals process. Selecting additional 
records after initial validation results are available would require an 
additional 2 to 3 months to select the sample, request the additional 
charts, provide sufficient time for hospitals to identify and send the 
requested charts, reabstract the data elements, and provide additional 
time for appeals processing. 

For the fiscal year (FY) 2007 APU determination, CMS will improve the 
stability of accuracy estimates by using all 4 quarters' validation 
estimates to provide a more stable estimate of their abstraction 
accuracy. As pointed out in the report, combining several quarters' 
accuracy estimates will provide a more stable estimate with lessened 
likelihood of clustering effects impacting sampling variability. This 
combining of multiple quarters' data will dramatically decrease the 
number of hospitals with statistically uncertain results for the FY 
2007 annual payment update determination. 

GAO Recommendation: 

Require hospitals to certify that they took steps to ensure that they 
submitted data on all eligible patients, or a representative sample 
thereof. 

CMS Response: 

As noted in the report, CMS assessed the completeness of hospital 
submitted data for Medicare beneficiaries in the FY 2005 and 2006 APIJ 
determinations. CMS found during the FY 2006 APU determination that 
about 20 percent of the prospective payment system hospitals submitted 
fewer patient records than claims submissions. CMS will develop 
stronger methods of assessing the completeness of hospital data 
submissions. We will implement two requirements: 

1. Require that hospitals formally attest to the completeness of their 
quarterly submission of quality data; and: 

2. Require that hospitals submit an aggregate count of all eligible 
Medicare and non-Medicare patients. 

GAO Recommendation: 

Assess the level of incomplete data submitted by hospitals for the APU 
program to determine the magnitude of underreporting, if any, in order 
to refine how completeness assessments may be done in future reporting 
efforts. 

CMS Response: 

As noted in the GAO report above, CMS assessed completeness by using 
Medicare claims data to compare to hospital submissions. 

As a result of implementing the two requirements described above, we 
will have data that enable us to assess completeness of hospital 
submissions for Medicare patients by comparing the number of 
submissions to the quality improvement organization clinical warehouse 
with both claims submissions and hospital-submitted eligible patient 
counts. For non-Medicare patients, the number of warehouse submissions 
will be compared with hospital-submitted patient counts. Hospitals will 
be asked to explain discrepancies. Based on this information, CMS will 
be able to consider whether it is necessary to take additional steps to 
assess and improve the completeness of submissions. 

The CMS believes that our methods for assessing abstraction accuracy 
and completeness of Medicare beneficiary submissions are basically 
sound. Again, we appreciate the insightful analysis and recommendations 
that GAO has provided CMS in this report. Among the steps that CMS will 
implement in improving our hospital reporting program are the 
following: 

* Combine several quarters of accuracy estimates to provide a more 
stable estimate of accuracy of chart abstraction; 

* Require that hospitals formally attest to the completeness of their 
quarterly submission of quality data; 

* Require that hospitals submit an aggregate count of all eligible 
Medicare and non-Medicare patients; 

* Analyze completeness of the hospital patient data submission by 
comparing submission counts with counts of claims submission and 
eligible individuals who are Medicare patients, and require hospitals 
to explain discrepancies; and: 

* Continue to provide quarterly feedback to hospitals about submission 
accuracy and completeness, and require them to explain discrepancies 
among counts. 

[End of section] 

Appendix V: GAO Contact and Staff Acknowledgments: 

GAO Contact: 

Cynthia A. Bascetta (202) 512-7101 or BascettaC@gao.gov: 

Acknowledgments: 

In addition to the contact named above, Linda T. Kohn, Assistant 
Director; Ba Lin; Nkeruka Okonmah; Eric A. Peterson; Roseanne Price; 
and Jessica C. Smith made key contributions to this report. 

FOOTNOTES 

[1] Pub. L. No. 108-173, § 501(b), 117 Stat. 2066, 2289-90 (amending 
section 1886(b)(3)(B) of the Social Security Act, to be codified at 42 
U.S.C. § 1395ww(b)(3)(B)). 

[2] The reduction in the annual payment update applies to hospitals 
paid under Medicare's inpatient prospective payment system. Critical 
access, children's, rehabilitation, psychiatric, and long-term-care 
hospitals may elect to submit data for any of the measures, but they 
are not subject to a reduction in their payment if they choose not to 
submit data. 

[3] Throughout this report, we refer to CMS's Reporting Hospital 
Quality Data for the Annual Payment Update program as the "APU 
program". 

[4] Throughout this report, we refer to the clinical data submitted by 
hospitals that are used to calculate their performance on the measures 
as "quality data". 

[5] Senate Bill 1932 would extend the APU program indefinitely. It 
would also increase the penalty for not submitting data to 2 percent 
and provide for the Secretary to establish additional measures, beyond 
the original 10, for payment purposes. 

[6] According to the Secretary of Health and Human Services, the effort 
is also intended to provide hospitals with a sense of predictability 
about public reporting expectations, to standardize data and data 
collection mechanisms, and to foster hospital quality improvement, in 
addition to providing information on hospital quality to the public. 

[7] For example, CMS plans to publicly report on the Hospital Compare 
Web site measures of patient perspectives on seven aspects of hospital 
care, with national implementation scheduled for 2006. 

[8] CMS's contractors for this program are the Iowa Foundation for 
Medical Care (IFMC) and DynKePRO, LLC. IFMC is the quality improvement 
organization (QIO) for the state of Iowa. (QIOs are independent 
organizations that work under contract to CMS to monitor quality of 
care for the Medicare program and help providers to improve their 
clinical practices.) Under a separate contract, IFMC operates the 
national database for hospital quality data known as the QIO clinical 
warehouse. DynKePRO, LLC, an independent medical auditing firm, 
operates CMS's Clinical Data Abstraction Center (CDAC), which assesses 
the accuracy of hospital data submissions. 

[9] Some hospitals contract with data vendors to electronically 
process, analyze, and transmit patient information. 

[10] Reabstraction is the re-collection of clinical data for the 
purpose of assessing the accuracy of hospital abstractions. In the APU 
program, CDAC compares data originally submitted by the hospitals to 
those it has reabstracted from the same medical records. 

[11] These were the calendar quarters for which, at the time we 
conducted our analysis, hospitals had collected the data and CMS had 
completed its process for reabstracting and assessing the data. We 
analyzed data for all hospitals affected by section 501(b) of MMA, 
which were located in 49 states and the District of Columbia. Hospitals 
in Maryland and Puerto Rico were excluded because they are paid under 
different payment systems than other acute care hospitals. 

[12] Throughout this report, we refer to this group of quality data 
reporting systems, each of which collects some type of clinical 
performance data from designated providers or health plans, as "other 
reporting systems". 

[13] The seven organizations were the American College of Cardiology, 
the California Office of Statewide Health Planning and Development, CMS 
(the units responsible for monitoring nursing home care regarding the 
Data Assessment and Verification Project contract), the Joint 
Commission on Accreditation of Healthcare Organizations (JCAHO), the 
National Committee for Quality Assurance, the New York State Department 
of Health, and the Society of Thoracic Surgeons. 

[14] HQA (formerly called the National Voluntary Hospital Reporting 
Initiative) was initiated by the American Hospital Association, the 
Federation of American Hospitals, and the Association of American 
Medical Colleges. It is supported by CMS, as well as the Joint 
Commission on Accreditation of Healthcare Organizations, National 
Quality Forum, American Medical Association, Consumer-Purchaser 
Disclosure Project, AARP, AFL-CIO, and Agency for Healthcare Research 
and Quality. Its aim is to provide a single standard quality measure 
set for hospitals to support public reporting and pay-for-performance 
efforts. 

[15] Throughout this report, we refer to the 10 measures on which 
reductions in the annual payment update are based as the "APU-measure 
set" and to the combination of those 10 with the additional measures 
adopted by HQA as the "expanded-measure set". HQA added 7 measures for 
discharges beginning April 1, 2004, and another 5 measures for 
discharges beginning July 1, 2004, for a total of 22 measures on which 
hospitals may currently submit data. Thus, the expanded-measure set 
includes different numbers of measures for different quarters of data. 

[16] The National Quality Forum is a voluntary standard-setting, 
consensus-building organization representing providers, consumers, 
purchasers, and researchers. 

[17] Patients under 18 years of age are excluded from the eligible 
patient population for the two cardiac conditions. 

[18] Before hospitals can consider sampling, rather than submitting all 
of their eligible cases, the number of eligible cases must exceed a 
minimum sample size that ranges from 60 per quarter for pneumonia cases 
to 76 for heart failure cases and 78 for heart attack cases. Once 
hospitals reach that threshold for a given condition, they can submit a 
random sample of their cases as long as the minimum sample size is met 
and it includes at least 20 percent of their eligible cases, up to a 
maximum sample size requirement of 241 for pneumonia, 304 for heart 
failure, and 311 for heart attacks. For discharges that occurred prior 
to January 1, 2005, CMS applied a different formula to hospitals not 
accredited by JCAHO that called for a minimum sample size of 7 for each 
of the three conditions and a sampling rate of at least 20 percent 
until a maximum sample size requirement of 70 cases was reached. 

[19] IFMC statistics show that a majority of hospitals ultimately 
succeed in gaining acceptance for all the cases they have submitted and 
that less than 10 percent of hospitals have had more than 5 percent of 
their cases rejected in a given quarter. 

[20] For two measures, influenza vaccination and prophylactic 
antibiotic selection for surgical patients, CMS has postponed public 
reporting. 

[21] DynKePRO, LLC, has operated CDAC since 1994. For 10 years it 
shared this function with a second firm, but in September 2004 DynKePRO 
negotiated a new contract with CMS that made it the sole CDAC 
contractor. In April 2005, DynKePRO became CSC York. 

[22] To be included in the reabstraction process, hospitals must have 
submitted data on at least six patients across all three conditions in 
that quarter. 

[23] The accuracy score is not based on all the data submitted by a 
hospital. Rather, CMS has identified a specific subset of the data 
elements that should be counted in computing the accuracy score. In 
general, CMS included in this subset the clinical data elements needed 
to calculate the hospital's rate for each of the measures and left out 
other administrative and demographic information about the patients. 
CMS estimates that five patient records usually contain about 100 data 
elements for calculation of the accuracy score, but the actual number 
of data elements depends on which conditions were involved and the 
number of measures for which a hospital submitted data. 

[24] Although CMS computes accuracy scores based on data for all 
measures submitted to the clinical warehouse, it recognizes that the 
MMA provision affecting hospital payments applies only to data for the 
10 measures specified for the APU program. See 69 Fed. Reg. 49080 (Aug. 
11, 2004). 

[25] CMS created an appeal process that allows a hospital to challenge 
the reabstraction results through its local QIO. For data from the 
first two calendar quarters of 2004, if the QIO agreed with the 
hospital's interpretation, the appeal was forwarded to CDAC for review 
and correction, if appropriate. CDAC's decision on the appeal was 
final. Beginning with data from the third calendar quarter of 2004, 
appealed cases no longer go back to CDAC. Instead, QIOs make the final 
decision to uphold either CDAC's or the hospital's interpretation. 
During this process, hospitals are not allowed to supplement the 
submitted patient medical records. 

[26] 70 Fed. Reg. 47420-47428 (Aug. 12, 2005). 

[27] CMS decided not to use accuracy scores from the first two quarters 
of the APU program because those data were collected before the 
alignment of CMS and JCAHO data collection specifications had begun to 
come into effect. Given the time needed to conduct all the steps in the 
process (see fig. 1), CMS was left with the third calendar quarter of 
2004 as the latest full quarter of data that could be used for 
determining the fiscal year 2006 update. The third calendar quarter 
also marked HQA's expansion to 22 measures. 

[28] Hospitals had to submit their patient medical records to CDAC for 
the fourth calendar quarter 2004 reabstractions no later than August 1, 
2005, to take advantage of this additional opportunity to pass the 80 
percent threshold. 

[29] The Hospital Compare Web site identifies instances where rates for 
a measure were based on fewer than 25 cases and where data were 
suppressed due to inaccuracies. However, the latter indication reflects 
situations where a hospital had problems with transmission of its data 
by a data vendor, not the outcome of the CDAC reabstractions. 

[30] Originally, CMS intended to apply JCAHO's sampling rules to JCAHO-
accredited hospitals, and its own sampling rules to the other 
hospitals, in computing their "expected cases". JCAHO's sampling 
procedures called for submitting larger samples to the clinical 
warehouse than CMS's did. However, when CMS officials determined that 
they could not reliably identify every hospital that belonged in the 
JCAHO group, they decided to apply the CMS rules across the board to 
all hospitals. Therefore, for many JCAHO-accredited hospitals, the 
number of "expected cases" computed by CMS underestimated the number of 
Medicare cases for which these hospitals should have submitted data, 
because JCAHO-accredited hospitals were to submit cases according to 
the JCAHO sampling rules. 

[31] Non-Medicare patients account for about 40 to 50 percent of all 
patients hospitalized for heart attacks and pneumonia and 20 to 32 
percent of those hospitalized for heart failure. For individual 
hospitals, these percentages could be higher or lower. 

[32] See appendix I for more detailed information on the limitations 
that applied to CMS's effort to estimate a minimum number of expected 
cases for each hospital. 

[33] The QualityNet Exchange Web site is the secure Internet connection 
used to transmit hospital quality data to the clinical warehouse. 

[34] For our analysis of baseline accuracy, the expanded-measure set 
includes the seven additional quality measures beyond the APU-measure 
set that HQA adopted for discharges after March 31, 2004. We found that 
some hospitals submitted data on the additional measures to the 
clinical warehouse for discharges occurring before that date, possibly 
because the hospitals were already collecting those data for JCAHO. 

[35] We assessed hospital capacity in terms of the number of patient 
beds. 

[36] For more detailed information on the relation of data accuracy to 
hospital characteristics and use of data vendors, see the tables in 
appendix III. 

[37] The data that we obtained from CMS specifically identified data 
vendors that JCAHO had certified for its own performance reporting 
system. These data vendors submitted data to the clinical warehouse for 
78 to 79 percent of the hospitals we analyzed for the two baseline 
quarters, while another 13 to 14 percent of hospitals directly 
submitted their own data. 

[38] Statistical uncertainty occurs because different samples generally 
produce different results, due to variation among the individual 
patients selected for different samples. With larger samples, 
differences in the results obtained from one sample to another 
decrease. Calculating a confidence interval provides a way to assess 
the effect of sample variation on the results obtained. Confidence 
intervals are usually computed at the 95 percent level. So if 100 
samples were selected, the result produced by 95 of them would likely 
fall between the low and high ends of the confidence interval. For 
example, one 300-plus-bed hospital in Virginia had an accuracy score of 
83.3 for the second calendar quarter of 2004 using the expanded-measure 
set, with a confidence interval that ranged from 76.8 to 89.9. There is 
a 95 percent likelihood that any sample selected for that hospital 
would generate an accuracy score that was greater than 76 and lower 
than 90. 

[39] The formula used to generate these confidence intervals takes into 
account variation in the number of individual data elements that were 
available in the five selected cases to compare the hospital's and 
CDAC's results. This is the same formula that is used by CMS, with one 
modification. Whereas CMS applied a one-tailed test at a 95 percent 
significance level to protect against hospitals receiving a failing 
score due to sampling error, we applied a two-tailed test at the 95 
percent significance level to identify both failing and passing scores 
that were statistically uncertain. (See app. I.) 

[40] Most, but not all, of the hospitals with statistically uncertain 
results had accuracy scores of 80 or above. See table 10 in appendix 
III. 

[41] For example, if a hospital had a confidence interval that ranged 
from 77 to 90, taking multiple samples would lead to some samples 
generating accuracy scores at or above 80 and other samples generating 
scores of less than 80. Whether that hospital passed the 80 percent 
accuracy threshold would depend on which of those samples was actually 
selected. 

[42] See 70 Fed. Reg. at 47422. 

[43] See appendix I for a more detailed description of this assessment. 

[44] For example, on-site auditors from one reporting system compare 
the data submitted against catheterization laboratory schedules and 
hospital billing records for the previous 12 months. Another reporting 
system hired a contractor to perform a one-time study comparing patient 
assessment data submitted by a facility against its total Medicare 
claims to identify instances where patient assessments were missing. 

[45] We have also published a document that describes a flexible 
framework for assessing data reliability, including both accuracy and 
completeness, when assessing computer-processed data. This document 
offers procedures that can be adapted to varying circumstances. These 
procedures include conducting electronic data testing, such as logic 
tests; ensuring internal control systems are in place that check the 
data when they are entered into the system and limit access to the 
system; checking for missing data elements as well as missing case 
records; and reviewing related documentation, which may include tracing 
a sample of records large enough to estimate an error rate back to 
their source documents. See GAO, Assessing the Reliability of Computer-
Processed Data, GAO-03-273G (Washington, D.C.: October 2002) External 
Version 1. 

[46] An official from one reporting system said that budgetary 
constraints limit the number of on-site audits that the system can 
perform. As a result, auditors from that system focus their review on 
hospitals with outcomes that fall above and below the systemwide 
average. 

[47] We downloaded various documents from the www.qnetexchange.org Web 
site between December 21, 2004, and January 10, 2006. 

[48] CMS included hospitals in Puerto Rico in its list of hospitals 
qualifying for the full fiscal year 2005 update, but determined in 
conjunction with the fiscal year 2006 payment update decision that 
Puerto Rico's hospitals were exempt from the APU program requirements. 
Hospitals in Puerto Rico receive prospective payments from Medicare, 
but under a different system than other hospitals. 

[49] The records we excluded were 536 surgery cases for the first 
quarter and 604 surgery cases for the second quarter, from hospitals 
providing data on surgical infection prevention measures. 

[50] IPRO, 2003 Review of Hospital Quality Reports for Health Care 
Consumers, Purchasers and Providers (Lake Success, N.Y.: October 2003); 
Delmarva Foundation and the Joint Commission on Accreditation of 
Healthcare Organizations, The State-of-the-Art of Online Hospital 
Public Reporting: A Review of Forty-Seven Websites (Easton, Md.: 
September 2004). 

GAO's Mission: 

The Government Accountability Office, the investigative arm of 
Congress, exists to support Congress in meeting its constitutional 
responsibilities and to help improve the performance and accountability 
of the federal government for the American people. GAO examines the use 
of public funds; evaluates federal programs and policies; and provides 
analyses, recommendations, and other assistance to help Congress make 
informed oversight, policy, and funding decisions. GAO's commitment to 
good government is reflected in its core values of accountability, 
integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains 
abstracts and full-text files of current reports and testimony and an 
expanding archive of older products. The Web site features a search 
engine to help you locate documents using key words and phrases. You 
can print these documents in their entirety, including charts and other 
graphics. 

Each day, GAO issues a list of newly released reports, testimony, and 
correspondence. GAO posts this list, known as "Today's Reports," on its 
Web site daily. The list contains links to the full-text document 
files. To have GAO e-mail this list to you every afternoon, go to 
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order 
GAO Products" heading. 

Order by Mail or Phone: 

The first copy of each printed report is free. Additional copies are $2 
each. A check or money order should be made out to the Superintendent 
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 
more copies mailed to a single address are discounted 25 percent. 
Orders should be sent to: 

U.S. Government Accountability Office 

441 G Street NW, Room LM 

Washington, D.C. 20548: 

To order by Phone: 

Voice: (202) 512-6000: 

TDD: (202) 512-2537: 

Fax: (202) 512-6061: 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: www.gao.gov/fraudnet/fraudnet.htm 

E-mail: fraudnet@gao.gov 

Automated answering system: (800) 424-5454 or (202) 512-7470: 

Public Affairs: 

Jeff Nelligan, managing director, 

NelliganJ@gao.gov 

(202) 512-4800 

U.S. Government Accountability Office, 

441 G Street NW, Room 7149 

Washington, D.C. 20548: