This is the accessible text file for GAO report number GAO-12-136 
entitled 'Health Care Quality Measurement: HHS Should Address 
Contractor Performance and Plan for Needed Measures' which was 
released on January 13, 2012. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as 
part of a longer term project to improve GAO products' accessibility. 
Every attempt has been made to maintain the structural and data 
integrity of the original printed product. Accessibility features, 
such as text descriptions of tables, consecutively numbered footnotes 
placed at the end of the file, and the text of agency comment letters, 
are provided but may not exactly duplicate the presentation or format 
of the printed version. The portable document format (PDF) file is an 
exact electronic replica of the printed version. We welcome your 
feedback. Please E-mail your comments regarding the contents or 
accessibility features of this document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

United States Government Accountability Office: 
GAO: 

Report to Congressional Committees: 

January 2012: 

Health Care Quality Measurement: 

HHS Should Address Contractor Performance and Plan for Needed Measures: 

GAO-12-136: 

GAO Highlights: 

Highlights of GAO-12-136, a report to congressional committees. 

Why GAO Did This Study: 

The Medicare Improvements for Patients and Providers Act of 2008 
(MIPPA) directed the Department of Health and Human Services (HHS) to 
enter into a 4-year contract with an entity to perform various 
activities related to health care quality measurement. In January 
2009, HHS awarded a contract to the National Quality Forum (NQF), a 
nonprofit organization that endorses health care quality measures-—
that is, recognizes certain ones as national standards. In 2010, the 
Patient Protection and Affordable Care Act (PPACA) established 
additional duties for NQF. This is the second of two reports MIPPA 
required GAO to submit on NQF’s contract with HHS. In this report—-
which covers NQF’s performance under the contract from January 14, 
2010, through August 31, 2011—-GAO examines (1) the status of projects 
under NQF’s required contract activities and (2) the extent to which 
HHS used or planned to use the measures it has received from NQF under 
the contract to meet its quality measurement needs, as of August 2011. 
GAO interviewed NQF and HHS officials, reviewed relevant laws, and 
reviewed HHS and NQF documents. 

What GAO Found: 

NQF has made progress on projects under its contract activities, as of 
August 2011. Specifically, NQF has completed or made progress on 60 of 
63 projects. For example, NQF has completed projects to endorse 
measures related to various topics, including nursing homes. However, 
for more than half of the projects, NQF did not meet or did not expect 
to meet the initial time frames approved by HHS. For example, NQF 
completed one project to retool measures—-that is, convert previously 
endorsed quality measures to an electronic format. While the retooling 
project was expected to be completed by September 2010, its completion 
was delayed by 3 months. NQF and HHS officials identified various 
reasons that contributed to this delay, including an expansion of the 
project’s scope and complexity. As a result of the delay, HHS did not 
have all the retooled measures it expected to include in its 
Electronic Health Records (EHR) Incentive Program. The delay of this 
project was also a contributing factor to NQF exceeding its estimated 
cost for its entire contract activity related to EHR by about $560,000 
in the second contract year—-January 14, 2010, through January 13, 
2011. While HHS monitored NQF’s progress through monthly progress 
reports and approved changes to time frames and costs, HHS did not use 
all of the tools for monitoring that are required under the contract. 
Specifically, HHS did not conduct an annual performance evaluation to 
assess timeliness and cost issues that could have helped to inform NQF’
s future scope of work. Until August 2011, HHS did not enforce the 
provision for NQF to submit a financial graph to compare monthly costs 
for each contract activity with cost estimates, which is information 
not included in monthly progress reports. These tools could have 
provided additional, more detailed information to help identify 
instances in which NQF might have been at risk of not meeting time 
frames or exceeding cost estimates, which could have provided HHS an 
opportunity to make any appropriate changes to NQF’s activities. 

HHS had used or planned to use about half of the measures—-164 of 344—-
that it received from NQF under the contract, as of August 2011. For 
example, HHS used 44 measures that NQF retooled under the contract in 
its EHR Incentive Program. HHS officials stated that the 44 measures 
used in the program contained errors, which required corrections. HHS 
officials also have not yet tested the retooled measures to assess the 
feasibility of implementing them in the electronic format; therefore, 
HHS runs the risk that some of these measures may not work as intended 
when implemented. HHS officials told GAO they expect to evaluate if 
and how they could use all of the remaining measures HHS received 
under the contract. However, HHS has not determined how PPACA 
requirements for quality measurement may have changed its needs for 
endorsed quality measures. As a result, HHS has not established a 
comprehensive plan that identifies its measurement needs and time 
frames for obtaining endorsed measures and that accounts for relevant 
PPACA requirements. Without such a plan, HHS may be limited in its 
efforts to prioritize which specific measures it needs to develop and 
to have endorsed by NQF during the remainder of the NQF contract. As a 
result, HHS may be unable to ensure that the agency receives the 
quality measures needed to meet PPACA requirements, including time 
frames for implementing quality measurement programs. 

What GAO Recommends: 

GAO recommends HHS: (1) use all monitoring tools required under the 
contract to help address NQF’s performance, (2) complete testing of 
retooled measures, and (3) comprehensively plan for its quality 
measurement needs. HHS neither agreed nor disagreed with these 
recommendations. NQF concurred with many of the findings in the report 
and provided additional context. 

View [hyperlink, http://www.gao.gov/products/GAO-12-136]. For more 
information, contact Linda T. Kohn at (202) 512-7114 or kohnl@gao.gov. 

[End of section] 

Contents: 

Letter: 

Background: 

NQF Made Progress on Projects under Each of Its Contract Activities 
but Has Not Met Expected Time Frames and Has Exceeded Cost Estimates: 

NQF Reported Costs and Fixed Fees of over $20 Million from January 
2010 through August 2011: 

HHS Used or Planned to Use Some NQF Measures Received under the 
Contract but Has Not Comprehensively Planned for Other Measurement 
Needs to Implement PPACA: 

Conclusions: 

Recommendations for Executive Action: 

Agency and Other External Comments and Our Evaluation: 

Appendix I: Health Care Quality Measurement Framework: 

HHS's Role in Planning and Identifying Gap Areas: 

Appendix II: How GAO Categorized NQF's 16 Tasks under the Contract 
into Nine Contract Activities: 

Appendix III: Status of NQF's Nine Contract Activities and the 
Projects under Each Activity GAO Identified: 

Appendix IV: Summary of Projects to Review the National Quality 
Forum's Endorsement Process: 

Appendix V: Comments from the Department of Health and Human Services: 

Appendix VI: Comments from the National Quality Forum: 

Appendix VII: GAO Contact and Staff Acknowledgments: 

Tables: 

Table 1: Description of the Nine NQF Contract Activities Categorized 
by GAO: 

Table 2: Status of Measures Used or Planned for Use under the NQF 
Contract, as of August 31, 2011: 

Table 3: GAO's Categorization of the NQF Contract's 16 Tasks into Nine 
Activities: 

Table 4: Status of Projects under NQF's Contract Activity to Provide 
Recommendations on a National Strategy and Priorities for Quality 
Measurement, as of August 31, 2011: 

Table 5: Status of Projects under NQF's Contract Activity to Endorse 
Health Care Quality Measures, as of August 31, 2011: 

Table 6: Status of Projects under NQF's Contract Activity to Maintain 
Previously Endorsed Health Care Quality Measures, as of August 31, 
2011: 

Table 7: Status of Projects under NQF's Contract Activity to Promote 
the Development and Use of Electronic Health Records, as of August 31, 
2011: 

Table 8: Status of Projects under NQF's Contract Activity to Submit an 
Annual Report to Congress and the Secretary of HHS, as of August 31, 
2011: 

Table 9: Status of Projects under NQF's Contract Activity to Provide 
Multistakeholder Input into HHS's National Strategy for Quality 
Improvement in Healthcare, as of August 31, 2011: 

Table 10: Status of Projects under NQF's Contract Activity to Provide 
Multistakeholder Input on Selection of Quality Measures, as of August 
31, 2011: 

Table 11: Status of Projects under NQF's Other Health Care Quality 
Measurement Activity Not Identified in MIPPA or PPACA, as of August 
31, 2011: 

Table 12: Status of Projects under NQF's Administrative Contract 
Activity: 

Figures: 

Figure 1: Estimated Time Frames and Completion Dates for All Five 
Projects under NQF's Endorsement Contract Activity, as of August 31, 
2011: 

Figure 2: Estimated Time Frames for Eight Projects Related to NQF's 
Contract Activity on the Promotion and Use of EHRs, as of August 31, 
2011: 

Figure 3: NQF's Reported Contract Activity Costs and Fixed Fees 
Totaling Approximately $12.8 Million during the Second Contract Year 
(Jan. 14, 2010--Jan. 13, 2011): 

Figure 4: NQF's Reported Contract Activity Costs and Fixed Fees 
Totaling Approximately $9.6 Million during the Period of the Third 
Contract Year Covered in Our Review (Jan. 14, 2011--Aug. 31, 2011): 

Figure 5: Health Care Quality Measurement Framework: 

Abbreviations: 

ASPE: Assistant Secretary for Planning and Evaluation: 

CHIP: Children's Health Insurance Program: 

CHIPRA: Children's Health Insurance Program Reauthorization Act of 
2009: 

CDP: consensus development process: 

CMS: Centers for Medicare & Medicaid Services: 

EHR: Electronic Health Record: 

ESRD: end-stage renal disease: 

FAR: Federal Acquisition Regulation: 

HAC: Healthcare Acquired Condition: 

HHS: Department of Health and Human Services: 

IT: Information Technology: 

IEP: Internal Evaluation Plan: 

LTCH: Long-Term Care Hospitals: 

MAP: Measure Applications Partnership: 

MAT: Measure Authoring Tool: 

Mathematica: Mathematica Policy Research, Inc. 

MIPPA: Medicare Improvements for Patients and Providers Act of 2008: 

NPP: National Priorities Partnership: 

NQF: National Quality Forum: 

National Quality Strategy: National Strategy for Quality Improvement 
in Healthcare: 

NTTAA: National Technology Transfer and Advancement Act of 1995: 

OMB: Office of Management and Budget: 

PPACA: Patient Protection and Affordable Care Act: 

PPS: Prospective Payment System: 

QDM: Quality Data Model: 

SRE: Serious Reportable Event: 

[End of section] 

United States Government Accountability Office: 
Washington, DC 20548: 

January 13, 2012: 

Congressional Committees: 

The nation's health care spending was about $2.5 trillion in 2009. 
Health care spending has been increasing in recent years by an average 
of nearly 7 percent per year since 2000, and is projected to reach 
over $4.5 trillion by 2020.[Footnote 1] However, higher levels of 
spending, such as spending more to increase the number or technical 
complexity of treatments, do not necessarily lead to a corresponding 
increase in the quality of care.[Footnote 2] To better align quality 
and health care spending, the Department of Health and Human Services 
(HHS) has developed various programs and initiatives to measure health 
care quality and to provide financial incentives to health care 
providers to improve quality and to reduce the use of unnecessary and 
costly services. These programs and initiatives include, for example, 
pay-for-performance programs that rely on information collected on 
various health care quality measures.[Footnote 3] Health care quality 
measures are used to evaluate how health care is delivered, and 
information obtained from such measures can promote accountability 
among health care providers. The Patient Protection and Affordable 
Care Act (PPACA), enacted in March 2010, directed HHS to expand its 
current programs and initiatives, such as pay-for-performance 
programs, and to implement new ones that will use health care quality 
measures. PPACA establishes specific time frames for HHS to implement 
these programs and initiatives that rely on various health care 
quality measures over the next several years. For example, PPACA 
requires HHS to establish a hospital value-based purchasing program. 
[Footnote 4] HHS estimates that in fiscal year 2013 its Hospital Value-
Based Purchasing Program[Footnote 5] will provide $850 million in 
Medicare incentive payments to hospitals that meet certain performance 
measures.[Footnote 6] Additionally, HHS is required under PPACA to 
establish value-based payments for physicians, develop a plan for 
value-based purchasing programs for skilled nursing home facilities 
and home health agencies, and create pilot programs that test the 
implementation of a value-based purchasing program for other 
providers--psychiatric, long-term care, inpatient rehabilitation, and 
cancer hospitals as well as hospice programs. 

For many of HHS's health care quality measurement programs and 
initiatives, HHS is required to give consideration to the quality 
measures endorsed by a consensus-based entity, such as the National 
Quality Forum (NQF). NQF is the entity in the United States with the 
lead responsibility for endorsing health care quality measures--that 
is, determining which measures should be recognized as national 
standards. The Medicare Improvements for Patients and Providers Act of 
2008 (MIPPA) directed the agency to enter into a 4-year contract with 
an entity to endorse health care quality measures and conduct other 
activities, and HHS contracted with NQF in January 2009.[Footnote 7] 
NQF is a nonprofit organization established in 1999 in order to foster 
agreement on national standards for measurement and public reporting 
of health care performance data. MIPPA provided $10 million per year 
for each of fiscal years 2009 through 2012--$40 million in total--from 
the Medicare Trust Funds for this 4-year contract, which covers the 
period from January 14, 2009, through January 13, 2013. PPACA provided 
an additional $100 million--$20 million per year for each of fiscal 
years 2010 through 2014--for additional health care quality 
measurement work to be conducted by NQF and HHS through fiscal year 
2014.[Footnote 8] Because PPACA does not specify the proportion of 
this amount for NQF activities, HHS officials told us that they plan 
to obligate approximately $10 million per year for the NQF contract, 
beginning in 2011, and that they may extend the contract through 
fiscal year 2014. 

MIPPA also directed us to submit two reports on the performance of and 
costs incurred by NQF under its 4-year contract with HHS.[Footnote 9] 
On July 14, 2010, we issued the first report, which contained 
information about NQF's performance and costs during the first 
contract year, which began January 14, 2009, and ended January 13, 
2010.[Footnote 10] This second report covers NQF's performance and 
costs for the second contract year--January 14, 2010, through January 
13, 2011--and part of the third contract year--January 14, 2011, 
through August 31, 2011. Specifically, this report examines (1) the 
status of projects under the activities NQF is required to complete 
under its contract with HHS, (2) the costs and fixed fees NQF reported 
under the contract from January 2010 through August 2011, and (3) the 
extent to which HHS used or planned to use the measures it receives 
from NQF under the contract to meet its quality measurement needs, as 
of August 31, 2011. In addition, appendix I of this report provides 
information about a framework for the various stages of a health care 
quality measure, including measure endorsement, as described by HHS 
and NQF officials and others. 

To describe the status of projects under NQF's required contract 
activities from January 2010 through August 2011, we reviewed relevant 
provisions in MIPPA and PPACA and documents including the HHS contract 
with NQF, NQF's 2009, 2010, and 2011 final annual work plans that 
cover the contract activities in response to MIPPA,[Footnote 11] NQF's 
2011 technical proposal for the contract activities in response to 
PPACA,[Footnote 12] and NQF's monthly progress and monthly internal 
evaluation plan reports that NQF is required to submit to HHS. 
[Footnote 13] For purposes of our work, we categorized the 16 tasks 
that NQF is required to perform under the contract from January 2010 
through August 2011 into 9 contract activities. (See appendix II for 
more detailed information on how we categorized these contract 
activities.) In addition, we identified the various projects NQF is 
required to perform under the 9 contract activities we identified. 
Specifically, for the purposes of our work, we identified 63 projects 
NQF is required to perform under the 9 contract activities, as shown 
in appendix III. We then compared the information we obtained from NQF 
on its progress on the 63 projects under the 9 activities to the 
requirements for NQF's performance under the contract with HHS and the 
expected time frames and cost estimates initially established by HHS 
and NQF in the final annual MIPPA work plans and other NQF documents 
for completing the work. We also interviewed NQF officials responsible 
for implementing the contract and HHS officials responsible for 
managing the contract and overseeing NQF's performance. We also 
compared evidence from HHS and NQF documents with evidence from 
discussions with HHS and NQF officials to corroborate any issues or 
challenges with NQF's performance. Based on our comparison and 
interviews with officials, we identified any changes to the work NQF 
is required to perform and changes to the time frames or cost 
estimates established in the final work plans developed annually to 
respond to MIPPA and a budget developed as part of NQF's technical 
proposal to respond to PPACA. 

To assess the costs and fixed fees NQF reported under the contract 
from January 2010 through August 2011, we reviewed NQF documents that 
include financial information related to the contract, such as NQF's 
final work plans developed in 2010 and 2011 to respond to MIPPA and a 
budget developed as part of NQF's technical proposal to respond to 
PPACA that describe NQF's estimated costs for each of the nine 
activities we identified under the contract. We obtained and 
summarized information from NQF monthly invoices submitted to HHS from 
January 14, 2010, through August 31, 2011, for direct[Footnote 14] and 
indirect[Footnote 15] costs incurred by NQF under the contract and 
fixed fees.[Footnote 16] We compared NQF's reported costs with its 
cost estimates for the activities in NQF's 2010 final annual work plan 
and determined whether or to what extent NQF's reported costs exceeded 
its cost estimates. We did not assess the accuracy of NQF's reported 
costs.[Footnote 17] We also reviewed the HHS contract with NQF and any 
contract modification since January 14, 2010. We also interviewed NQF 
and HHS officials responsible for reviewing and approving the costs 
and fixed fees submitted under the contract. Based on our review of 
documents and interviews with NQF and HHS officials, we determined 
that the reported costs and fixed-fee data were sufficiently reliable 
for the purposes of this report. 

To examine the extent to which HHS used or planned to use measures it 
receives from NQF under the contract to meet its quality measurement 
needs, as of August 2011, we reviewed statutory requirements and 
documents related to HHS health care quality measurement efforts. We 
relied on HHS to identify programs and initiatives that use or will 
use health care quality measures, such as programs or initiatives 
established by MIPPA, PPACA, and the Health Information Technology for 
Economic and Clinical Health Act.[Footnote 18] As a result, the health 
care quality programs and initiatives mentioned in our report may not 
represent a comprehensive list of all those that use or plan to use 
measures received from NQF under the contract. We interviewed 
officials from various divisions within HHS that may use measures that 
were nationally recognized--or NQF-endorsed--in their various programs 
and initiatives related to health care quality measurement in order to 
corroborate the information we obtained from documentation. We 
obtained information about how HHS selects measures for its programs 
and initiatives from our interviews with HHS officials and reviews of 
HHS documents. We determined how HHS has used and planned to use 
measures that were received under the contract with NQF by identifying 
all the measures that HHS received under the contract and matching 
each measure to the quality measures being used or considered for use 
in HHS programs and initiatives.[Footnote 19] For the purpose of this 
report, we defined a measure as used if the measure was identified by 
HHS and included in an HHS program or initiative as of August 31, 
2011, and indicated as such by a final rule or other HHS 
documentation. We defined a measure as planned for use if the measure 
was identified by HHS for potential use in an HHS program or 
initiative as of August 31, 2011, and indicated as such by a proposed 
rule or other HHS documentation. In addition, we reviewed HHS 
documents and interviewed HHS officials to determine whether or to 
what extent HHS has plans for using work conducted under the NQF 
contract to meet PPACA requirements related to quality measures for 
HHS programs and initiatives. For example, we reviewed HHS plans or 
strategies such as the 2011 National Strategy for Quality Improvement 
in Healthcare (National Quality Strategy),[Footnote 20] that describe 
the agency's plans to help meet its quality measurement needs. 

We conducted this performance audit from April 2011 through January 
2012 in accordance with generally accepted government auditing 
standards. Those standards require that we plan and perform the audit 
to obtain sufficient, appropriate evidence to provide a reasonable 
basis for our findings and conclusions based on our audit objective. 
We believe that the evidence obtained provides a reasonable basis for 
our findings and conclusions based on our audit objectives. 

Background: 

NQF is a nonprofit organization established in 1999 in order to foster 
agreement, or consensus, on national standards for measuring and 
public reporting of health care performance data.[Footnote 21] Its 
membership includes more than 400 organizations that represent 
multiple sectors of the health care system, including providers, 
consumers, and researchers. NQF's mission focuses on three core areas: 
(1) building consensus on national priorities and goals for 
performance improvement and working in partnership to achieve them, 
(2) endorsing national consensus standards for measuring and publicly 
reporting on performance, and (3) promoting the attainment of national 
goals through education and outreach programs. 

Prior to its contract with HHS, NQF established a consensus 
development process (CDP) to evaluate available health care quality 
measures to determine which ones are qualified to be endorsed--that 
is, recognized--as national standards. Under this process, 
organizations that develop quality measures submit them to NQF for 
consideration, in response to specific solicitations by NQF.[Footnote 
22] NQF forms a committee of experts from its member organizations as 
well as other organizations and agencies to conduct an objective and 
transparent review of these quality measures against four standardized 
criteria established by NQF, such as whether the measures are 
scientifically acceptable.[Footnote 23] After this committee evaluates 
the measures against these criteria, NQF's process allows for a period 
during which its member organizations and the public may comment on 
the committee's recommendation for each measure. The process also 
provides for a period for its member organizations to vote on whether 
the measures should be endorsed by NQF as a national standard. 
Ultimately, NQF's board of directors makes a final decision on whether 
NQF should formally endorse the measures. 

As of October 2011, NQF has endorsed over 600 health care quality 
measures in 27 areas, such as cancer and diabetes. HHS uses NQF-
endorsed measures in its programs and initiatives to promote quality 
measurement, and NQF continues to endorse quality measures separate 
from its contract with HHS. 

NQF's Contract with HHS: 

NQF's work under the contract includes endorsement of quality measures 
and other activities that are expected to support HHS's quality 
measurement efforts, such as through value-based purchasing programs. 
Specifically, NQF's work under the contract consists of various 
projects under the nine contract activities related to health care 
quality measurement. The work plans developed annually to respond to 
MIPPA and NQF's technical proposal to respond to PPACA delineate the 
projects NQF is required to conduct under the nine contract 
activities, as well as expected time frames and cost estimates for the 
projects for each year. Table 1 provides more detailed information on 
the nine contract activities. Some of these activities are required by 
either MIPPA or PPACA, while others are quality measurement activities 
established by HHS or administrative activities. 

Table 1: Description of the Nine NQF Contract Activities Categorized 
by GAO: 

Contract activities required by MIPPA or PPACA: 

NQF contract activity categorized by GAO: Recommendations on a 
National Strategy and Priorities for Quality Measurement (as required 
by MIPPA); 
Description of activity: Contract activities required by MIPPA or 
PPACA: NQF will synthesize evidence and convene key stakeholders to 
make recommendations on a national strategy and priorities for health 
care performance measurement. 

NQF contract activity categorized by GAO: Endorsement of Health Care 
Quality Measures (as required by MIPPA); 
Description of activity: Contract activities required by MIPPA or 
PPACA: NQF will conduct endorsement of quality measures using its 
consensus development process to determine which ones should be 
endorsed as national standards. Specifically, it will provide for the 
endorsement of quality measures through projects that focus on 
priority areas identified by HHS, such as patient outcomes and patient 
safety. In addition to endorsing measures, this activity includes an 
evaluation of NQF's endorsement process to identify ways to improve 
its efficiency and effectiveness. 

NQF contract activity categorized by GAO: Maintenance of Endorsed 
Quality Measures (as required by MIPPA); 
Description of activity: Contract activities required by MIPPA or 
PPACA: NQF will use its process to ensure that endorsed measures are 
maintained--that is, updated or retired. This process is similar to 
NQF's endorsement process, in that it involves a review of measures 
against established criteria, a period for public comment, and a final 
decision by NQF's board of directors.[A] Under this activity, NQF will 
conduct maintenance projects using its 3-year review cycle that 
compares previously endorsed measures against new measures submitted 
for endorsement. In addition to the 3-year review cycles, NQF can also 
maintain measures through: (1) annual updates, which is a process 
where NQF annually reviews and receives information on a measure to 
determine whether the measure has undergone any changes; (2) ad hoc 
reviews where NQF reviews an endorsed measure at any time based on 
justifiable evidence to substantiate the review; and (3) time-limited 
endorsement reviews, which is a process where NQF will review time-
limited endorsed measures--measures that have been endorsed for a 
limited period of time to allow for testing by measure developers. 

NQF contract activity categorized by GAO: Promotion of the Development 
and Use of Electronic Health Records (as required by MIPPA); 
Description of activity: Contract activities required by MIPPA or 
PPACA: NQF will undertake projects to promote the development and use 
of electronic health records for use in quality measurement, including 
to "retool" measures--that is, convert previously endorsed quality 
measures to an electronic format that is compatible with electronic 
health records. In addition, NQF will undertake projects related to 
developing tools to capture data for performance measurement, creating 
a framework that defines a standardized set of data that should be 
captured in patients' electronic health records, and providing data to 
providers to support clinical decisions. These efforts had not been 
performed by NQF prior to the contract. 

NQF contract activity categorized by GAO: Annual Report to Congress 
and the Secretary of Health and Human Services (as required by MIPPA 
and amended by PPACA); 
Description of activity: Contract activities required by MIPPA or 
PPACA: NQF is required to submit an annual report to Congress and the 
Secretary of HHS that provides a summary of NQF's progress on 
activities under the contract during the year. PPACA established 
additional requirements that NQF must perform for this activity that 
include providing additional information in its annual report to 
Congress and the Secretary of HHS, beginning with its report due March 
1, 2012. Specifically, NQF will include additional information in the 
report on gaps in endorsed and nonendorsed health care quality 
measures and a summary of activities conducted by multistakeholder 
groups during the third contract year (January 14, 2011 - January 13, 
2012). 

NQF contract activity categorized by GAO: Multistakeholder Input into 
HHS's National Strategy for Quality Improvement in Health Care (as 
required by PPACA); 
Description of activity: Contract activities required by MIPPA or 
PPACA: In 2010, NQF convened a multistakeholder group to provide input 
on national priorities for improvement in population health and in the 
delivery of health care services for consideration under HHS's 
National Strategy for Quality Improvement in Healthcare, or the 
National Quality Strategy.[B] In 2011, NQF will continue to engage a 
multistakeholder group to provide input on a plan for measuring and 
improving health and health care focusing on the six priorities 
identified in HHS's 2011 National Quality Strategy.[C] In addition, 
NQF will conduct endorsement projects for cross-cutting measures[D] 
that can support national implementation of these priorities. 

NQF contract activity categorized by GAO: Multistakeholder Input on 
the Selection of Quality Measures for use in Payment Programs and 
Value-Based Purchasing Programs Under PPACA, Other Private/Public 
Payers, and Other Programs (as required by PPACA); 
Description of activity: Contract activities required by MIPPA or 
PPACA: NQF will convene a multistakeholder group to provide input on 
the selection of measures for use in various public programs, 
including measures for use in quality reporting programs for 
physicians, postacute care programs under Medicare, Prospective 
Payment System (PPS) exempt cancer hospitals,[E] and hospice care. In 
addition, the group will provide input on measures that address 
quality issues identified for vulnerable populations, such as dual-
eligible beneficiaries. 

Other health care quality measurement activities not identified in 
MIPPA or PPACA: 

NQF contract activity categorized by GAO: Other health care quality 
measurement activity; 
Description of activity: Contract activities required by MIPPA or 
PPACA: NQF will work on additional health care quality measurement 
projects established by HHS to fill immediate areas of need, including 
two white papers related to resource use and efficiency; efficiency 
and resource use endorsement projects; and projects related to measure 
harmonization[F]. 

Administrative activities: 

NQF contract activity categorized by GAO: Administrative activity; 
Description of activity: Contract activities required by MIPPA or 
PPACA: NQF will perform a number of administrative activities that 
focus on project planning and contract management: (1) conduct an 
opening meeting between HHS and NQF, (2) develop an annual work plan, 
(3) develop a monthly internal evaluation plan for quality assurance, 
(4) develop a monthly progress report, (5) conduct weekly conference 
calls, and (6) develop a public website[G]. 

Source: GAO analysis of MIPPA, PPACA, and the NQF contract as well as 
the 2010 and 2011 NQF final annual MIPPA work plans and 2011 PPACA 
technical proposal. 

[A] Prior to its contract with HHS, NQF established a process for 
maintenance of measures. In May 2010, NQF updated this process by 
requiring all previously endorsed measures to go through a 
comprehensive review every 3 years that will include a comparison 
against newly submitted measures submitted for endorsement. For 
example, NQF will review all previously endorsed and newly submitted 
measures related to cardiovascular health simultaneously. 

[B] NQF convened the National Priorities Partnership to complete this 
work. 

[C] The six priorities identified in the National Quality Strategy 
are: (1) making care safer; (2) ensuring person-and family-centered 
care; (3) promoting effective communication and coordination of care; 
(4) promoting the most effective prevention and treatment of the 
leading causes of mortality, starting with cardiovascular disease; (5) 
working with communities to promote wide use of best practices to 
enable healthy living; and (6) making quality care more affordable. 

[D] Cross-cutting measures are those that affect all or most patients, 
such as patient safety. 

[E] PPS-exempt cancer hospitals are those exempt from the Prospective 
Payment System (PPS), which is a system of Medicare reimbursement in 
which providers receive a predetermined, fixed payment based on the 
service provided and its classification, such as the patient's 
diagnosis at the time of hospital admission. 

[F] Harmonization is a process where related measures are standardized 
in a manner by which they can be used in multiple settings when 
relevant. NQF defines "related measures" as those that either (1) 
address the same concepts for measure focus (target process, 
condition, event, outcome) but a different target population; or (2) 
address different concepts for measure focus (target process, 
condition, event, outcome) but the same target population. 

[G] Development of a public website was part of the 2010 final annual 
MIPPA work plan but this work was canceled in 2011. 

[End of table] 

To help determine the activities and the projects under the nine 
contract activities that NQF is expected to perform during each 
contract year, HHS has established an interagency workgroup that 
comprises officials from multiple divisions within HHS, including the 
Agency for Healthcare Research and Quality, the Centers for Medicare & 
Medicaid Services (CMS), and the Office of the National Coordinator 
for Health Information Technology. The workgroup is responsible for 
prioritizing and selecting the activities and projects under each 
activity that NQF is expected to perform during each contract year. 
[Footnote 24] HHS officials told us that the representatives from 
these various HHS agencies provide input on the work NQF is expected 
to perform, including determining quality measures requested from NQF 
for their respective programs. The activities and projects selected by 
the interagency workgroup become part of NQF's scope of work under the 
contract.[Footnote 25] Some of the projects under the contract 
activities that NQF is expected to perform during the year will be 
ongoing from the previous contract year while new work will be 
incorporated into the work plan as necessary. 

For the NQF contract, HHS selected a cost-plus-fixed-fee contract, 
under which HHS reimburses NQF for actual costs incurred under the 
contract in addition to a fixed fee that is unrelated to costs. Cost-
plus-fixed-fee contracts are used for efforts such as research, 
design, or study efforts where costs and technical uncertainties exist 
and it is desirable to retain as much flexibility as possible in order 
to accommodate change.[Footnote 26] However, this type of contract 
provides only a minimum incentive to the contractor to control costs. 
As we reported in 2009, these contracts are suitable when the cost of 
work to be done is difficult to estimate and the level of effort 
required is unknown.[Footnote 27] This cost-plus-fixed-fee contract is 
NQF's first cost-reimbursement contract. For cost-reimbursement 
contracts, the Federal Acquisition Regulation (FAR)[Footnote 28] 
requires appropriate government surveillance[Footnote 29] during 
performance to provide reasonable assurance that efficient methods and 
effective cost controls are used. Under the FAR, contracts are to 
contain a provision for agency approval of a contractor's 
subcontracts.[Footnote 30] HHS's contract with NQF contains this 
provision and also requires the approval of consultants engaged under 
the contract. The review and approval of NQF's use of subcontractors 
and consultants require appropriate support documentation provided by 
NQF to HHS, including a description of the services, the proposed 
price, and a negotiation memo that reflects the principal elements of 
the price negotiations between NQF and the subcontractor or 
consultant. Under its contract with HHS, NQF has utilized 31 
subcontractors and 16 consultants since January 14, 2010, to provide 
support to NQF on many of the contract activities and associated 
projects. 

Two HHS components are principally responsible for administering the 
NQF contract: the office of the Assistant Secretary for Planning and 
Evaluation (ASPE) and CMS--an agency within HHS.[Footnote 31] 
Specifically, the project officer for the NQF contract is a 
representative of ASPE.[Footnote 32] This individual is responsible 
for program management and works with the contracting officer to 
oversee the contract. The contracting officer for the NQF contract, 
responsible for administering the contract, is a representative of 
CMS.[Footnote 33] The contract outlines various activities that the 
contracting officer is required to perform, such as conducting an 
annual evaluation of the contractor's performance. 

NQF Made Progress on Projects under Each of Its Contract Activities 
but Has Not Met Expected Time Frames and Has Exceeded Cost Estimates: 

From January 14, 2010, through August 31, 2011, NQF made progress on 
projects under its contract activities. However, our review of NQF 
documents found that NQF had not met or did not expect to meet time 
frames on more than half of the projects, and it exceeded its cost 
estimates for projects under three of the contract activities. HHS did 
not use all tools for monitoring that are required under the contract. 

NQF Made Progress on Projects under Each of the Contract Activities as 
of August 31, 2011: 

From January 14, 2010, through August 31, 2011, NQF has made progress 
on 60 of the 63 projects under the activities required under its 
contract with HHS. Specifically, NQF had completed 26 projects and was 
continuing to work on the remaining 34 projects.[Footnote 34] 
(appendix III provides the status of all contract activities and the 
projects under each activity NQF was expected to perform during our 
reporting period.) Examples of projects under the contract activities 
include both completed and continuing projects: 

* Endorsement of Measures Activity. NQF endorsed 101 measures since 
the beginning of the second contract year by conducting work on 
endorsement projects on different topic areas.[Footnote 35] 
Specifically, NQF completed two projects to endorse 38 outcome 
measures related to 20 high-priority conditions identified by CMS that 
account for the majority of Medicare's costs,[Footnote 36] and mental 
health and child health conditions; and 21 performance measures for 
chronic and postacute care nursing facilities. NQF also worked on two 
projects related to child health quality and patient safety. As of 
August 2011, NQF endorsed 41 child health quality measures and 1 
patient safety measure under these projects. NQF expected to complete 
the child health quality project in September 2011 and the patient 
safety project in December 2011.[Footnote 37] In addition, NQF 
completed a contractually required review of its endorsement process, 
subcontracting with Mathematica Policy Research, Inc. (Mathematica). 
The review focused on the timeliness and effectiveness of the 
endorsement process;[Footnote 38] identified inefficiencies, including 
those that may contribute to delays; and recommended, among other 
steps, that NQF create a schedule for its endorsement process for 
measure developers and develop feasible time lines that include clear 
goals for each endorsement project. HHS officials stated that 
Mathematica's recommendations were valuable because much of the work 
under the NQF contract needs to be completed in an accelerated 
timeline to help fill critical measurement gaps associated with HHS's 
health care quality programs and initiatives. For more information 
about this review, see appendix IV. 

* Maintenance of Endorsed Quality Measures Activity. NQF maintained--
that is, updated or retired--124 measures under the contract since the 
beginning of the second contract year.[Footnote 39] These included 41 
measures reviewed under NQF's 3-year review cycle related to diabetes, 
mental health, and musculoskeletal conditions.[Footnote 40] In 
addition, 83 measures were maintained under NQF's other maintenance 
review processes.[Footnote 41] NQF was also continuing to work on 
maintenance projects it initiated in 2010 for measures related to 
cardiovascular and surgery measures. As of August 2011, the two 
projects were expected to be completed by December 2011 and January 
2012, respectively.[Footnote 42] 

* Promotion of the Development and Use of Electronic Health Records 
Activity. NQF has made progress on three projects related to 
retooling--that is, converting previously endorsed quality measures to 
an electronic format that is compatible with electronic health records 
(EHR). First, NQF completed initial retooling of 113 measures. This 
work is intended to allow data from EHRs to be used for quality 
measurement, which is a part of HHS's long-term goal to use health 
information technology to exchange information and improve quality of 
care. Second, as of August 2011, NQF convened an expert review panel 
to review the retooled measures to ensure that each retooled measure 
is properly formatted, the logic is correctly stated, and the intent 
of the measures is maintained in the electronic format that will use 
data obtained from EHRs, instead of from claims as originally 
formatted. Third, as of August 2011, NQF was expected to complete 
another project to provide an updated list of the 113 retooled 
measures to HHS by December 2011, which would incorporate any 
revisions identified by the expert review panel and others involved in 
the retooling process. After these updated measures are completed, HHS 
officials told us that they will contract with other entities to 
conduct testing of some of the 113 retooled measures to assess the 
feasibility of implementing the measures in the electronic format. 
Although NQF's endorsement process requires that measure developers 
submit data on validity and reliability testing of measures they 
submit for endorsement, this testing does not include feasibility 
testing for implementing the measures in an electronic format for 
performance measurement. As of December 2011, HHS officials did not 
provide an expected date of completion for this feasibility testing 
but told us that they have awarded two contracts that include this in 
their scope of work.[Footnote 43] In addition to the retooling 
projects, NQF is developing a software tool--the Measure Authoring 
Tool--to allow measure developers to create standardized electronic 
measures that help capture information in EHRs so that less retooling 
would be needed in the future. As of August 2011, NQF was completing 
final testing of the beta, or initial, version of this tool. NQF 
expected to complete testing and publish an updated version for public 
use by January 2012.[Footnote 44] 

* Multistakeholder Input into HHS's National Quality Strategy 
Activity. NQF convened the National Priorities Partnership (NPP), a 
multistakeholder group expected to provide annual input on national 
priorities, among other things, to be considered in the National 
Quality Strategy.[Footnote 45] As of August 2011, the NPP was 
completing a report on this input, which was then published in 
September 2011. The report noted the need for a national comprehensive 
strategy that identifies core sets of standardized measures to meet 
each of the national priorities HHS identified in the 2011 National 
Quality Strategy, among other things. The NPP noted in the report that 
a common data platform, core measure set, and public reporting 
mechanism are key components of the infrastructure for performance 
measurement. It also highlighted that a strategic plan, road map, and 
timeline for establishing an infrastructure should be accelerated to 
allow for rapid implementation over the next 5 years. Additionally, 
the NPP reported that it was critical that all federal programs drive 
toward the establishment of a common platform for measurement and 
reporting. 

* Multistakeholder Input on the Selection of Quality Measures 
Activity. NQF has convened the Measure Applications Partnership (MAP). 
The MAP is a multistakeholder group that is expected to conduct work 
in two areas. First, the MAP is expected to provide input to the 
Secretary of HHS on the selection of quality measures for use in 
payment programs and value-based purchasing programs required by 
PPACA, among others. The MAP will review a list of measures published 
by the Secretary of HHS on December 1 of each year, and develop a 
report that contains a framework to help guide measure selection. The 
MAP will provide its annual input beginning February 1, 2012, for 
measures used in the following 11 programs: hospice, hospital 
inpatient, hospital outpatient, physician offices, cancer hospitals, 
end-stage renal disease (ESRD) facilities, inpatient rehabilitation 
facilities, long-term care hospitals, hospital value-based purchasing, 
psychiatric hospitals, and home health care. Second, the MAP is 
expected to publish reports that provide input on the selection of 
measures for use in various quality reporting programs, including 
those for physicians. As of August 2011, the MAP had held meetings and 
initiated its work for reports due October 1, 2011.[Footnote 46] 

* Other Health Care Quality Measurement Activity. NQF completed a 
project to endorse six imaging efficiency measures. NQF was also 
continuing to work on a project to identify existing quality measures 
and gap areas related to measurement of regionalized emergency care 
services.[Footnote 47] 

NQF Did Not Meet Expected Time Frames and Exceeded Cost Estimates: 

Our review of NQF documents found that NQF had not met or did not 
expect to meet time frames on more than half of the projects under the 
contract activities that were completed or ongoing, as of August 2011. 
Specifically, our review of documents found that NQF had not met 
expected time frames on 18 of the 26 projects it completed under the 
nine contract activities. Further, NQF did not expect to meet time 
frames on 14 of the 34 projects on which it was continuing to work. 
The delays of these projects under the contract activities varied in 
time from about 1 to 12 months. HHS officials told us they approved 
all changes to the time frames, which were established by HHS and NQF 
in NQF's 2010 and 2011 final annual MIPPA work plans and the PPACA 
technical proposal. Appendix III provides the status for all projects 
related to each of the nine contract activities, including information 
on their expected and actual time frames for completion during our 
reporting period. Examples of projects under the contract activities 
for which NQF did not meet or did not expect to meet expected time 
frames include the following: 

* Endorsement of Measures Activity. NQF did not meet or did not expect 
to meet time frames for all five endorsement projects under the 
endorsement contract activity.[Footnote 48] (See appendix III for 
details on the five projects.) For example, NQF was expected to 
complete an endorsement project for nursing home quality measures in 
July 2010; however, the measures were not endorsed until February 
2011. (See figure 1 for estimated time frames and actual completion 
dates for all projects related to the endorsement contract activity.) 

Figure 1: Estimated Time Frames and Completion Dates for All Five 
Projects under NQF's Endorsement Contract Activity, as of August 31, 
2011: 

[Refer to PDF for image: timeline] 

Endorsement project: 

Patient outcomes: 

Phases I & II: 
Original project time frame: August, 2009-September, 2010; 
Actual project time frame: August 2009-March 2011[A]; 
Completion of key interim steps in project: 
Report 1: August 2010; 
Report 2: January 2011; 
Report 3: March 2011; 
Actual start date: August 2009; 
Actual/expected completion date: March 2011[A]. 

Phase III (Child health, mental health): 
Original project time frame: August, 2009-September, 2010; 
Actual project time frame: August 2009-January 2011; 
Actual start date: August, 2009; 
Actual/expected completion date: January 2011. 

Patient safety: 

Phase I: 
Original project time frame: September 2009-January 2011; 
Actual project time frame: September 2009-December 2011 (expected); 
Actual start date: September 2009; 
Actual/expected completion date: Expected December 2011. 

Phase II: 
Original project time frame: September 2009-January 2011; 
Actual project time frame: September 2009-September 2011 (expected)[B]; 
Actual start date: September 2009; 
Actual/expected completion date: September 2011 (expected). 

Nursing home measures:  

Original project time frame: September 2009-July 2010; 
Actual project time frame: September 2009-February 2011; 
Actual start date: September 2009; 
Actual/expected completion date: February 2011. 

Evaluation of CDP process[C]: 

Original project time frame: October 2009-July 2010; 
Actual project time frame: October 2009-April 2011 (key step completed 
in January 2011); 
Actual start date: October 2009; 
Actual/expected completion date: April 2011. 

Child health quality: 

Original project time frame: May 2010-April 2011; 
Actual project time frame: May 2010-September 2011 (expected)[D]; 
Actual start date: May 2010; 
Actual/expected completion date: September 2011 (expected). 

Source: GAO analysis of NQF and HHS data. 

Note: For purposes of our report, we consider an endorsement project 
completed when the board ratifies its measures. 

[A] NQF had three separate reviews of measures during Phases I and II, 
resulting in three separate board endorsements. The circles reflect 
those three reviews. 

[B] NQF endorsed one out of two patient safety measures under phase II 
of the patient safety project. The remaining measure was still under 
consideration by the board and is scheduled to be reviewed for 
endorsement on September 15, 2011. 

[C] Mathematica's report on its evaluation of NQF's CDP process, or 
the endorsement process, and NQF's response to Mathematica's 
recommendations in the report were published in January 2011. NQF 
solicited comments on its proposed revisions and additions to its 
endorsement process, as a result of the Mathematica evaluation, 
through April 2011. 

[D] NQF endorsed 41 out of 44 child health quality measures on August 
15, 2011. As of August 2011, the remaining 3 measures were still under 
consideration by the board and were scheduled to be reviewed for 
endorsement on September 15, 2011. 

NQF officials stated that several factors contributed to NQF exceeding 
the expected time frames for the five endorsement projects, including 
the high volume of measures submitted for review, the amount of time 
it took to harmonize measures[Footnote 49] between measure developers, 
and a need for additional technical expertise on review panels. 

[End of figure] 

* Promotion of the Development and Use of Electronic Health Records 
Activity. NQF did not meet or did not expect to meet expected time 
frames for five out of eight projects related to the EHR contract 
activity.[Footnote 50] For example, NQF was expected to complete its 
initial retooling of 113 endorsed quality measures into electronic 
formats by September 2010, but this effort was not completed until 
December 2010. (See figure 2 for estimated time frames and actual 
completion dates for all projects related to the EHR contract 
activity.) In addition, NQF was expected to complete the project to 
convene an expert panel to review the 113 retooled measures by January 
2011. However, the panel did not complete its review of the 113 
measures until June 2011. 

Figure 2: Estimated Time Frames for Eight Projects Related to NQF's 
Contract Activity on the Promotion and Use of EHRs, as of August 31, 
2011: 

[Refer to PDF for image: timeline] 

EHR project: 

Measure Authoring Tool (MAT): 

Original project time frame: September 2009-December, 2010; 
Actual project time frame: September 2009-January 2012 (expected)[A]; 
Completion of key interim steps in project: 
Beta version: January 2011; 
Actual start date: September 2009; 
Actual/expected completion date: January 2012 (expected)[A]. 

Clinical decision support: 

Original project time frame: July 2009-September, 2010; 
Actual project time frame: July 2009-December 2010; 
Actual start date: July 2009; 
Actual/expected completion date: December 2010. 

eMeasure creation and testing: 

Original project time frame: February 2009-February 2010; 
Actual project time frame: February 2009-April 2010[B]; 
Completion of key interim steps in project: 
Report: April 2010; 
Actual start date: February 2009; 
Actual/expected completion date: April 2010[B]. 

Quality Data Model (QDM): 

Original project time frame: September 2009-ongoing, through February 
2012; 
Actual project time frame: September 2009-ongoing, through February 
2012; 
Completion of key interim steps in project: 
Version 2.0: June 2010; 
Version 2.1: September 2010; 
Version 3.0: My 2011; 
Version 2012: January 2012; 
Actual start date: September 2009; 
Actual/expected completion date: Ongoing. 

Health information technology utilization expert panel: 

Original project time frame: October 2009-October 2010; 
Actual project time frame: October 2009-December 2010; 
Actual start date: October 2009; 
Actual/expected completion date: December 2010. 

Initial retooling of 113 measures: 

Original project time frame: February 2009-September 2010; 
Actual project time frame: February 2009-December 2010[D]; 
Completion of key interim steps in project: 
44 of 113 retooled: July 2010; 
Actual start date: February 2009; 
Actual/expected completion date: December 2010[D]. 

Update of 113 retooled measures: 

Original project time frame: February 2011-December 2011; 
Actual project time frame: February 2011-December 2011 (expected); 
Actual start date: February 2011; 
Actual/expected completion date: December 2011 (expected). 

eMeasure Review Panel: 

Original project time frame: April 2010-January 2011; 
Actual project time frame: March 2010-October 2011 (expected)[E]; 
Completion of key interim steps in project: 
All reviewed: July 2011; 
Actual start date: March 2010; 
Actual/expected completion date: October 2011 (expected)[W]. 

Source: GAO analysis of NQF and HHS data. 

[A] In January 2011, NQF completed a version of the tool that was 
released for beta testing (reflected by the circle). NQF is now 
updating the version, which it expects to release in January 2012. 

[B] NQF officials told us that the final report was completed in April 
2010; however, additional work was required to complete the project 
through the beginning of July 2010. 

[C] NQF is expected to publish several versions of the QDM throughout 
the contract term to allow for ongoing review and updates. The circles 
indicate the versions that have been published or are expected to be 
published, as of August 2011. 

[D] NQF was unable to complete all 113 retooled measures by September 
2010, as initially scheduled. NQF submitted versions of 44 measures in 
July 2010, as requested by HHS. The final versions of all 113 measures 
were not completed until December 2010. 

[E] NQF officials told us the eMeasure Review Panel was not able to 
complete its review of the retooled measures by its initial due date 
of January 13, 2011, because all 113 measures were not retooled until 
December 2010. As a result, the review was not completed until June 
2011. As of August 2011, the review panel was expected to submit its 
final report in October 2011. NQF officials told us in October 2011 
this report was completed, as scheduled. 

[End of figure] 

According to HHS and NQF officials, several factors contributed to NQF 
exceeding expected time frames for the retooling project under the EHR 
contract activity. HHS officials stated that the first set of 44 
retooled measures submitted had errors that required correction. For 
example, HHS officials stated that they found errors in the electronic 
coding of these 44 retooled measures requiring NQF and its 
subcontractors who retooled the measures to make corrections. In 
addition, HHS and NQF officials stated that after starting the 
retooling project, they quickly learned that the estimated time frames 
for the retooling project, as well as other projects related to the 
EHR activity, were overly ambitious, given the scope and complexity of 
the work. For example, HHS officials noted that retooling of quality 
measures into electronic format had never been attempted before and 
the technical complexity and labor required to complete the project 
were greater than anticipated. NQF officials also told us that HHS's 
requests to modify the scope of work for this project often required 
changing the time frame for completing the retooled measures. These 
factors resulted in an extension of the project that delayed the final 
delivery of the 113 retooled measures as well as contributed to the 
need for additional staff at NQF. 

* Other Health Care Quality Measurement Activity. NQF was expected to 
complete two projects under the other health care quality measurement 
activity related to efficiency and resource use--one white paper on 
resource use[Footnote 51] and another on geographic-level efficiency 
[Footnote 52] by July 2010. These white papers were intended to 
provide information for an endorsement project on resource-use 
measures that began in January 2011. However, as of August 2011, the 
resource-use paper was still under review by HHS, and NQF officials 
stated they expected to receive comments in September 2011.[Footnote 
53] The geographic-level efficiency paper was canceled in June 2010 at 
the request of HHS. NQF initially intended to subcontract the work on 
these two projects, but officials told us that they were unable to 
identify a subcontractor at the level of funding approved for this 
project. As a result, HHS approved NQF's proposal to complete this 
work internally. HHS officials stated that the drafts NQF submitted on 
both topics were poor in quality and did not meet its needs, resulting 
in HHS requesting additional revisions for the resource-use white 
paper that delayed its completion, and requesting the cancellation of 
the geographic-level efficiency white paper. 

* Administrative Activity. NQF did not meet the expected time frames 
for completing one of the required projects under the administrative 
activity--finalizing its annual work plan. Specifically, the NQF 
contract requires NQF to develop an annual work plan and to receive 
final approval from HHS within the first 4 weeks of each contract 
year; however, NQF did not meet this requirement in 2010 or 2011. For 
example, the final 2011 MIPPA annual work plan was not developed by 
NQF and approved by HHS until April 1, 2011. According to NQF and HHS 
officials, the 2011 MIPPA work plan was not developed and approved on 
time due to extended discussions on the scope and cost estimates of 
NQF's EHR activities. HHS officials told us that the primary reason 
for the extended discussions was that they expected the costs to 
reflect all the work needed to complete the Measure Authoring Tool 
(MAT) by the end of the second contract year. However, they said that 
NQF only submitted a beta version of the tool by the end of the second 
contract year, which was not the version expected by HHS. NQF 
officials told us that the version was never intended to be final but 
rather a beta version, consistent with their understanding of HHS's 
expectations. As a result, HHS and NQF officials needed to evaluate 
the scope of work and cost estimates for this and other projects. 
Further, NQF officials told us the delay in completing the 2011 MIPPA 
annual work plan resulted in the interruption of NQF's ongoing work 
related to the MAT under the EHR contract activity. The delay also 
delayed its receipt of funding for some new or ongoing work under the 
contract. In some instances, NQF chose to start new or continue 
ongoing work with its own funding. For example, NQF officials stated 
that NQF began work related to the MAP using its own funds until HHS 
authorized the work. In addition, the delay in completing the 2011 
MIPPA work plan resulted in the need to set the start date for fall 
2011 rather than earlier in the contract year for some of the projects 
under the maintenance activity. 

NQF also exceeded its cost estimates for projects under three of the 
contract activities. HHS officials told us they approved the changes 
to the cost estimates and in some cases modified NQF's scope of work 
to help ensure that NQF's costs did not exceed the amount HHS had 
obligated for the contract activities. NQF officials stated that in 
certain cases, not meeting expected time frames contributed to NQF 
exceeding these cost estimates. For example, the delays in projects 
related to the EHR contract activity, including expanding the scope of 
the retooling project, contributed to NQF exceeding its cost estimate 
of about $3.8 million for the entire EHR contract activity by about 
$560,000 in the second contract year.[Footnote 54] In another example, 
the delays in finalizing the 2010 and 2011 MIPPA work plans 
contributed in part to NQF exceeding its cost estimate for developing 
and finalizing these plans, which is a project under the 
administrative contract activity. Specifically, while NQF estimated 
that completion of the annual work plan would cost approximately 
$77,000, NQF reported an actual cost of $176,590. In addition, NQF 
also exceeded its cost estimate for the endorsement contract activity 
during the second contract year for various reasons, including a need 
for additional technical experts for review panels. Specifically, NQF 
exceeded estimated costs of about $3.1 million for the entire 
endorsement activity by about $146,000 in the second contract year. 
[Footnote 55] While HHS officials told us they approved all changes to 
the cost estimates, in certain cases they reduced the scope of NQF's 
work in 2011 to ensure that total available funding for the contract 
year was not exceeded and that sufficient funding was available for 
ongoing projects. For example, HHS officials told us that they had 
hoped to start several new endorsement projects beginning in 2011; 
however, these projects were not included in the 2011 final annual 
MIPPA work plan so that funding would be available for NQF to complete 
its ongoing projects, including work that was delayed under the EHR 
contract activity. In addition, HHS requested that NQF discontinue its 
work on one project related to the development of a public website for 
2011, which is associated with the administrative contract activity. 

HHS Did Not Use All Monitoring Tools Required under the NQF Contract: 

HHS officials told us that to help monitor NQF's performance on the 
projects under the contract activities, they rely on NQF to report any 
issues, including those related to time frames or cost estimates, in 
the monthly progress reports that NQF is required to submit to HHS or 
during phone calls held at least monthly. While HHS monitored NQF's 
progress and approved changes to the time frames and cost estimates 
for the projects under the contract activities, HHS did not use 
available tools for monitoring that are required under NQF's contract. 
These tools could have helped to provide an opportunity for HHS to 
make any appropriate changes to NQF's projects. For example, HHS did 
not conduct an annual performance evaluation required by the contract 
that would assess timeliness and cost control issues, among other 
things, for the previous contract year.[Footnote 56] The results of 
such an evaluation could help HHS officials to consider potential 
timeliness and cost issues when determining NQF's scope of work for 
the next year. Further, while monthly progress reports and invoices 
include information on NQF's costs, these documents do not compare 
reported costs to initial cost estimates. HHS officials told us that, 
prior to August 2011, they had not enforced a contractual requirement 
for NQF to submit--nor had it received from NQF--a financial graph in 
its monthly progress reports that provides information comparing NQF's 
monthly incurred costs for each of the contract activities with 
initial cost estimates. Instead, HHS officials informally requested 
that NQF provide them with the financial status of the contract 
activities in midyear 2010, which helped them to plan for NQF's work 
under the contract for 2011. Having a financial graph in the monthly 
progress report could have helped HHS officials to identify instances 
where any contract activity was approaching or exceeding NQF's initial 
cost estimates prior to HHS's midyear review. This, in turn, could 
have provided HHS and NQF an opportunity to adjust estimates of future 
costs for these or related activities earlier in the contract year. 
HHS officials had asked NQF to begin to include such a financial graph 
in its monthly progress reports beginning in August 2011. 

NQF Reported Costs and Fixed Fees of over $20 Million from January 
2010 through August 2011: 

From January 14, 2010, through August 31, 2011, NQF reported a total 
of approximately $22.4 million in costs and fixed fees on monthly 
invoices submitted to HHS for projects under activities conducted in 
response to MIPPA and PPACA. Specifically, NQF reported about $12.8 
million in total costs and fixed fees for the contract activities it 
performed during the second contract year--January 14, 2010, through 
January 13, 2011.[Footnote 57] From January 14, 2011, through August 
31, 2011, part of the third contract year, NQF reported an additional 
$9.6 million in total costs and fixed fees.[Footnote 58] 

During the second contract year, the majority of NQF's reported costs 
were related to the promotion of the development and use of EHRs (36 
percent, or $4.6 million) and endorsement of health care quality 
measures (26 percent, or $3.3 million). Figure 3 illustrates the costs 
and fixed fees NQF reported for eight of the nine contract activities 
we reviewed that occurred during the second contract year. The ninth 
contract activity relates to multistakeholder input on the selection 
of quality and efficiency measures, as directed by PPACA. This 
contract activity did not begin until after January 14, 2011, which is 
the start of the third contract year. 

Figure 3: NQF's Reported Contract Activity Costs and Fixed Fees 
Totaling Approximately $12.8 Million during the Second Contract Year 
(Jan. 14, 2010--Jan. 13, 2011): 

[Refer to PDF for image: pie-chart] 

Promotion of the development and use of EHRs: 36% ($4.66 million); 
Endorsement of health care quality measures: 26% ($3.32 million); 
Recommendations on a national strategy and priorities for quality 
measurement: 10% ($1.35 million); 
Other health care quality measurement activity: 10% ($1.23 million); 
Administrative activity: 10% ($1.02 million); 
Maintenance of endorsed health care quality measures: 6% ($0.73 
million); 
Fixed fees: 3% ($0.36 million); 
Multistakeholder input into HHS's 2011 National Quality Strategy: less 
than 1% ($0.10 million). 
Annual report to Congress and the Secretary of HHS: less than 1% 
($0.09 million). 

Source: GAO analysis of NQF data. 

Note: The figure describes eight of the nine contract activities we 
reviewed. 

[End of figure] 

For the part of the third contract year covered in our review--January 
14, 2011, through August 31, 2011--almost one-half of NQF's reported 
costs were for the activity to promote the development and use of EHRs 
and for the activity to provide multistakeholder input on the 
selection of quality and efficiency measures. Each of these activities 
accounted for 22 percent, or about $2.1 million of NQF's reported 
costs. Other costs reported by NQF include those for the activity 
related to providing multistakeholder input into HHS's annual National 
Quality Strategy ($1.55 million, or 16 percent) and those for the 
activity related to the maintenance of endorsed quality measures 
activity (13 percent, or $1.29 million). Figure 4 illustrates the 
costs and fixed fees NQF reported for the part of the third contract 
year covered in our review for each of the nine contract activities we 
reviewed. 

Figure 4: NQF's Reported Contract Activity Costs and Fixed Fees 
Totaling Approximately $9.6 Million during the Period of the Third 
Contract Year Covered in Our Review (Jan. 14, 2011--Aug. 31, 2011): 

[Refer to PDF for image: pie-chart] 

Multistakeholder input on the selection of quality and efficiency 
measures: 22% (2.14 million; 
Promotion of the development and use of EHRs: 22% ($2.10 million); 
Multistakeholder input into HHS's National Quality Strategy: 16% 
($1.55 million); 
Maintenance of endorsed health care quality measures: 13% ($1.29 
million); 
Other health care quality measurement activity: 10% ($0.97 million); 
Endorsement of health care quality measures: 6% ($0.60 million); 
Fixed fees: 4% ($0.41 million); 
Recommendations on a national strategy and priorities for quality 
measurement: 3% ($0.31 million); 
Administrative activity: 1% ($0.10 million); 
Annual report to Congress and the Secretary of HHS: less than 1% 
($0.067 million). 

Source: GAO analysis of NQF data. 

[End of figure] 

According to HHS, as of August 2011, about $55.2 million remains 
available for the NQF contract.[Footnote 59] About $15.1 million in 
MIPPA funding remains available for work to be conducted through 
January 13, 2013. In addition, HHS plans to obligate approximately 
$40.1 million of its PPACA funding through 2014 for NQF's activities 
related to health care quality measurement in response to PPACA. 
[Footnote 60] 

HHS Used or Planned to Use Some NQF Measures Received under the 
Contract but Has Not Comprehensively Planned for Other Measurement 
Needs to Implement PPACA: 

For its various programs or initiatives, HHS has used or planned to 
use about one-half of the quality measures that NQF has endorsed, 
maintained, or retooled under the contract, as of August 31, 2011, and 
HHS officials expect to evaluate if and how the remaining measures 
will be used. However, HHS has not comprehensively determined how it 
will use NQF's work under the contract to implement PPACA requirements 
related to quality measures. 

HHS Has Used or Planned to Use 164 of the 344 Measures That NQF Has 
Endorsed, Maintained, or Retooled under the Contract: 

According to HHS officials, HHS has used or planned to use[Footnote 
61] about one-half (164) of the 344 health care quality measures it 
has received from NQF through various endorsement, maintenance, and 
retooling projects under the contract, as of August 31, 2011.[Footnote 
62] For example, of the 164 measures used or planned for use, 44 were 
used in CMS's Medicare and Medicaid EHR Incentive Program after being 
retooled--that is, converted to an electronic format that is 
compatible with EHRs--under the NQF contract.[Footnote 63] Although 
these 44 retooled measures were used in the EHR Incentive Program, HHS 
officials stated that NQF and HHS detected coding and other errors in 
the versions of the 44 retooled measures that were published in the 
program's final rule in July 2010 that required NQF to make 
corrections to them after publication of the final rule. NQF did not 
submit the revised versions of the 44 retooled measures published in 
the final rule to HHS until December 2010. HHS officials stated that 
because the final rule had already been published prior to receiving 
the final formatted measures, CMS listed general guidance on its 
website to address the errors. HHS officials told us that these 44 
measures are being used but have not yet been tested to assess the 
feasibility of implementing them in the electronic format.[Footnote 
64] Until the testing is complete, HHS runs the risk that some of 
these measures may not work as intended when implemented in electronic 
format for performance measurement. As a result, the agency does not 
have reasonable assurance that the retooled versions of the measures 
will correctly capture information from EHRs. In addition to the 44 
retooled measures used in the EHR Incentive Program, HHS also has used 
or planned to use 120 measures that it received from endorsement and 
maintenance projects under the NQF contract for various HHS programs 
and initiatives. (See table 2 for details on specific programs in 
which HHS has used or planned to use health care quality measures 
received from NQF under the contract.) 

Table 2: Status of Measures Used or Planned for Use under the NQF 
Contract, as of August 31, 2011: 

Promote the development and use of electronic health records: 

Projects: Measure retooling; 
Total measures received from NQF under the contract: 113; 
Measures used or planned for use by HHS: 44; 
Description of HHS programs and initiatives for which NQF measures are 
being used or planned for use: Forty-four measures have been used in 
Stage I of the Electronic Health Records (EHR) Incentive Program[A]. 

Total retooled measures: 
Total measures received from NQF under the contract: 113; 
Measures used or planned for use by HHS: 44. 

Endorse health care quality measures: 

Projects: Patient outcomes; 
Total measures received from NQF under the contract: 38; 
Measures used or planned for use by HHS: 6; 
Description of HHS programs and initiatives for which NQF measures are 
being used or planned for use: Six measures are planned for use in 
various programs including the Hospital Inpatient Quality Reporting 
Program[B]. 

Projects: Patient safety; 
Total measures received from NQF under the contract: 1; 
Measures used or planned for use by HHS: 0; 
Description of HHS programs and initiatives for which NQF measures are 
being used or planned for use: No measures used or planned for use. 

Projects: Nursing homes; 
Total measures received from NQF under the contract: 21; 
Measures used or planned for use by HHS: 17; 
Description of HHS programs and initiatives for which NQF measures are 
being used or planned for use: Seventeen measures are planned for use 
in various programs including Nursing Home Compare[C]. 

Projects: Child health quality; 
Total measures received from NQF under the contract: 41; 
Measures used or planned for use by HHS: 8; 
Description of HHS programs and initiatives for which NQF measures are 
being used or planned for use: Eight measures have been used in CMS 
Medicaid Programs as part of CMS's Children's Health Insurance Program 
Reauthorization Act of 2009 (CHIPRA) Initial Core Set of Children's 
Health Care Quality Measures[D]. 

Total endorsed measures: 
Total measures received from NQF under the contract: 101; 
Measures used or planned for use by HHS: 31. 

Maintain previously endorsed health care quality measures[E]: 

3-Year maintenance review cycle[F]: 

Projects: Diabetes; 
Total measures received from NQF under the contract: 13; 
Measures used or planned for use by HHS: 10; 
Description of HHS programs and initiatives for which NQF measures are 
being used or planned for use: Twenty-three measures have been used or 
are planned for use in various programs and initiatives including the 
Physician Quality Reporting System[G] and the Hospital Outpatient 
Quality Reporting Program[H]. 

Projects: Mental health; 
Total measures received from NQF under the contract: 11; 
Measures used or planned for use by HHS: 3; 
Description of HHS programs and initiatives for which NQF measures are 
being used or planned for use: Twenty-three measures have been used or 
are planned for use in various programs and initiatives including the 
Physician Quality Reporting System[G] and the Hospital Outpatient 
Quality Reporting Program[H]. 

Projects: Musculoskeletal conditions; 
Total measures received from NQF under the contract: 17; 
Measures used or planned for use by HHS: 10; 
Description of HHS programs and initiatives for which NQF measures are 
being used or planned for use: Twenty-three measures have been used or 
are planned for use in various programs and initiatives including the 
Physician Quality Reporting System[G] and the Hospital Outpatient 
Quality Reporting Program[H]. 

Other maintenance review processes: 

Projects: Ad hoc review; 
Total measures received from NQF under the contract: 3; 
Measures used or planned for use by HHS: 2; 
Description of HHS programs and initiatives for which NQF measures are 
being used or planned for use: Two measures have been used or are 
planned for use in various programs, including the EHR Incentive 
Program and the Hospital Inpatient Quality Reporting Program. 

Projects: Time-limited review; 
Total measures received from NQF under the contract: 80; 
Measures used or planned for use by HHS: 63; 
Description of HHS programs and initiatives for which NQF measures are 
being used or planned for use: Sixty-three measures have been used or 
are planned for use in various programs and initiatives, including the 
EHR Incentive Program and the Physician Quality Reporting System. 

Total maintained measures: 
Total measures received from NQF under the contract: 124; 
Measures used or planned for use by HHS: 88. 

Other health care quality measurement activity: 

Projects: Imaging efficiency measures; 
Total measures received from NQF under the contract: 6; 
Measures used or planned for use by HHS: 1; 
Description of HHS programs and initiatives for which NQF measures are 
being used or planned for use: One measure is planned for use in the 
Hospital Outpatient Quality Reporting Program. 

Total measures from other quality measurement activity: 
Total measures received from NQF under the contract: 6; 
Measures used or planned for use by HHS: 1. 

Total measures: 
Total measures received from NQF under the contract: 344; 
Measures used or planned for use by HHS: 164. 

Source: GAO analysis of final rules, proposed rules, and other HHS 
documentation. 

Note: Measures that are endorsed, maintained, and retooled under the 
NQF contract are counted separately, even if they are the same 
measure. Therefore, the 344 measures HHS has received from NQF do not 
represent 344 distinct measures. For example, a measure that had been 
maintained under the contract and later retooled would be counted as 
two separate measures. Therefore, the measure would appear twice in 
our count of the total number of measures received from NQF--once as a 
maintained measure and then separately as a measure that was retooled. 
Similarly, the measure would also be counted as two separate measures 
for purposes of determining the number used or planned for use. If 
both versions of the measure were used or planned for use in HHS 
programs or initiatives, then it would also appear twice in our count 
of measures used or planned for use. 

[A] The EHR Incentive Program provides incentive payments to eligible 
Medicare and Medicaid providers that adopt and use health information 
technology (IT) and reduces payments for those Medicare professionals 
or hospitals that do not. 

[B] The Hospital Inpatient Quality Reporting program provides 
hospitals a financial incentive to report the quality of their 
services as well as provides CMS data to help consumers make more 
informed decisions about their health care. 

[C] Nursing Home Compare is a tool of the Nursing Home Quality 
Initiative that allows consumers, providers, states, and researchers 
to compare information on health quality measures among various 
nursing homes. 

[D] CMS's CHIPRA Initial Core Set of Children's Health Care Quality 
Measures identifies an initial core set of child health quality 
measures for voluntary use by state Medicaid and Children's Health 
Insurance Program (CHIP) programs to evaluate the quality of care and 
health outcomes in children enrolled in Medicaid and CHIP programs. 

[E] NQF can also maintain measures through its (1) annual updates, (2) 
ad hoc requests, and (3) time-limited endorsement review processes. 
NQF officials told us that as of August 2011, 83 measures have been 
maintained using these processes, including 80 under time-limited 
review and 3 under ad hoc request. 

[F] NQF officials told us the review of the diabetes, mental health, 
and musculoskeletal measures were maintained under its previous 3-year 
maintenance review process because the project began in September 
2009, which is prior to the updated process that took effect in August 
2010. 

[G] The Physician Quality Reporting System provides an incentive 
payment to eligible professionals who select among 240 measures to 
report. 

[H] CMS's Hospital Outpatient Quality Reporting is a pay-for-
performance program for outpatient hospital services. 

[End of table] 

HHS officials told us that they expect to evaluate if and how they 
could use all of the remaining 180 of the 344 quality measures that 
were endorsed, maintained, or retooled under the NQF contract that are 
not currently in use or planned for use in HHS programs or 
initiatives. According to HHS officials, any measure developer can 
submit a measure to be considered for NQF endorsement. Therefore, all 
the measures received under the contract may not be applicable to a 
particular HHS health care quality program or initiative. HHS 
officials told us that they will review the remaining 180 measures to 
determine if they are applicable to their health care quality programs 
or initiatives. The officials expect that many of these measures will 
be used in HHS programs or initiatives required by PPACA. For example, 
HHS officials told us that they will consider implementation of most 
of the retooled measures in future stages of the EHR Incentive 
Program. In addition, PPACA directed HHS to establish a hospital value-
based purchasing program, as well as to make plans or begin pilot 
programs for value-based purchasing in other settings of care. The 
hospital value-based purchasing program will use various quality 
measures and depend on the information collected on them to determine 
payments to providers. PPACA also required the development of no less 
than 10 provider-level outcome measures for hospitals and physicians 
by March 2012. Further, PPACA directed HHS to identify quality 
measures that could be used to evaluate hospice programs and publish 
these measures by October 1, 2012. HHS officials told us that they are 
in the process of determining whether or to what extent the remaining 
180 measures HHS has received under the NQF contract can be used to 
address the new measurement needs and priorities established by PPACA. 
HHS officials told us that they prefer to use NQF-endorsed measures to 
meet HHS's measurement needs because these quality measures are 
nationally recognized standards and in some cases HHS is required to 
use them.[Footnote 65] 

HHS Does Not Have a Comprehensive Plan for Determining How It Will Use 
NQF's Work under the Contract to Implement PPACA Requirements Related 
to Quality Measurement: 

Although HHS has taken steps to determine how it can use the measures 
received under the contract with NQF, the agency does not have a 
comprehensive plan for determining how it will use the remainder of 
the work conducted under NQF's contract to implement PPACA 
requirements, including plans for additional quality measures that 
need to be endorsed during the remaining contract years. HHS officials 
told us that HHS determines on an annual basis which activities--
including work on quality measures--NQF is to perform under the 
contract through the interagency workgroup. The workgroup is comprised 
of representatives from various HHS agencies and allows them to 
provide input on their needs, including quality measures that need 
endorsement from NQF, for their respective programs. However, HHS 
officials told us that each HHS program assesses its quality 
measurement needs separately and provides varying levels of detail 
about its needs. Therefore, it is unclear the extent to which all 
programs consistently incorporate PPACA's quality measurement 
requirements and deadlines into these assessments. The NPP's September 
2011 report noted the importance of greater alignment of national 
quality measurement efforts, including the establishment of a 
comprehensive measurement strategy that identifies core measure sets, 
among other things. In addition, the report noted that all federal 
programs should work toward the establishment of a common platform for 
measurement and reporting. Without a comprehensive plan that 
delineates HHS's quality measurement needs, and given that each 
program assesses its quality measurement needs separately, the 
interagency workgroup may not be able to systematically ensure that 
all of HHS's quality measurement needs that implement PPACA 
requirements align with the selection and prioritization of activities 
for NQF to complete under the contract. 

While HHS has begun various efforts to assess its quality measurement 
needs, the lack of a plan that comprehensively determines the impact 
of PPACA on its needs could affect the agency's progress on its 
quality measurement efforts as well as how it selects and prioritizes 
NQF's contract activities. Officials told us that prior to PPACA's 
enactment, CMS maintained a 5-year plan that listed its measurement 
needs based on agency priorities and the priorities established by the 
NPP for some of its programs.[Footnote 66] However, officials told us 
that they have not updated this list to reflect the requirements 
related to quality measurement and time frames established by PPACA. 
In March 2011, HHS published the National Quality Strategy as required 
by PPACA, which included six priority areas of focus. The report was 
required by PPACA to include agency-specific plans, goals, benchmarks, 
and standardized quality metrics for each priority area, but did not 
do so. HHS officials stated that this document describes HHS's initial 
plan for these elements and that they may be included in future 
versions of the strategy. In June 2011, HHS officials told us that 
they plan to convene a Quality Measurement Task Force within CMS with 
a goal to comprehensively align, coordinate, and approve the 
development, maintenance, and implementation of health care quality 
measures for use in various CMS programs. As of August 2011, the task 
force was in an early stage of development, and therefore it is too 
early to determine whether it will accomplish its goal. Although these 
various HHS efforts are key steps toward helping the agency meet its 
quality measurement needs, they are not guided by a comprehensive plan 
that synthesizes key priority areas identified in various sources, 
such as those reported by the NPP or in the National Quality Strategy, 
for which measures may be needed. Without such a plan, HHS may be 
limited in its efforts to prioritize which specific measures it needs 
to develop and have endorsed by NQF for its health care quality 
programs and initiatives established by PPACA. As a result, HHS may be 
unable to ensure that the agency receives the quality measures needed 
to meet PPACA requirements and specified time frames related to 
quality measurement. 

Conclusions: 

Health care quality measures are increasingly important to HHS as it 
uses and will continue to use them in its existing and forthcoming 
programs and initiatives to evaluate health care delivery. For 
example, HHS's value-based purchasing programs are pay-for-performance 
programs that will require providers to collect and report information 
on health care quality measures and adjust payment levels based on 
providers' performance against the measures. PPACA has increased HHS's 
quality measurement needs, and the time frames specified in the law 
have also increased the urgency of obtaining endorsed quality 
measures--which are nationally recognized standards and in some cases 
are required by statute--to meet these needs. Given that NQF is the 
entity in the United States with lead responsibility for endorsing 
health care quality measures, NQF's endorsement activities under the 
contract are of key importance to help meet HHS's quality measurement 
needs. 

However, NQF's endorsement process takes time. For more than half of 
the projects, including all five projects in the endorsement activity, 
NQF did not meet or did not expect to meet the initial time frames 
approved by HHS. In addition, projects under three of the contract 
activities have exceeded initial cost estimates, which resulted in 
HHS's modification of NQF's scope of work in some instances to help 
ensure that NQF's costs did not exceed the funding allocated for the 
contract activities. While HHS received information in monthly 
progress reports to help monitor NQF's performance under the contract, 
the agency did not use all of the monitoring tools required under the 
contract to help address issues related to time lines and cost 
estimates. These monitoring tools included an annual performance 
evaluation that could help HHS officials consider potential issues 
related to NQF's time frames and cost estimates when planning work for 
the next year and a financial graph to be included in NQF's monthly 
progress reports. The graph would have compared reported costs to 
initial cost estimates, which is something that monthly progress 
reports do not do. Although HHS officials reported that they recently 
began in August 2011 to enforce the contractual requirement for NQF to 
submit the graph, they have not implemented the required annual 
performance evaluation. By not taking advantage of these tools, HHS 
runs the risk of not having detailed and timely information that could 
help identify instances in which NQF might be at risk of not meeting 
time frames or exceeding estimated costs. Identifying such instances 
could provide an opportunity for HHS to make any appropriate changes 
to NQF's scope of work, including setting priorities to ensure that 
HHS receives the quality measures it needs in a timely manner. 

With the time remaining under the contract, HHS has an opportunity to 
ensure that the work performed under NQF's contract better meets the 
agency's needs for its programs and initiatives. However, HHS has not 
developed a plan that comprehensively identifies its quality 
measurement needs for its programs and initiatives in light of PPACA's 
requirements or determines how it will use the work conducted during 
the remaining years of the NQF contract to help it meet these needs. 
In addition, critical tasks may need to be completed outside of the 
NQF contract. For example, HHS requested that NQF retool 113 measures 
under the contract and used 44 of the 113 measures that included 
errors in its EHR Incentive Program. As of November 2011, feasibility 
testing related to implementation of the retooled measures had not 
been completed, and HHS expected to perform this work outside of the 
NQF contract. Until the testing is completed, HHS runs the risk that 
some of the retooled measures may not work as intended when 
implemented in electronic format for performance measurement, which is 
a concern because use of these measures is an important component of 
HHS's long-term goal for providers to use health information 
technology (IT) to exchange information and improve the quality of 
care. 

Without a comprehensive plan, HHS lacks assurance that its selection 
of the work to be performed by NQF--and the approximately $55.2 
million that the agency expects to spend for remaining work under the 
NQF contract--will be prioritized in the most effective way possible. 
Given that PPACA includes time frames for the implementation of 
quality measurement programs, NQF's pace in completing some of the 
work under the contract--particularly the endorsement activity--raises 
concerns. If the endorsement projects continue to require extended 
completion times, HHS runs the risk of not having all the endorsed 
measures it needs for implementing its programs and initiatives. 
Should this occur, HHS may need to select nonendorsed measures for its 
programs and initiatives that have not undergone an objective and 
transparent review by NQF. 

Recommendations for Executive Action: 

To help ensure that HHS receives the quality measures it needs to 
effectively implement its quality measurement programs and initiatives 
within required time frames, we recommend that the Secretary of HHS 
take the following three actions: 

* use monitoring tools required under the NQF contract to obtain 
detailed and timely information on NQF's performance and use that 
information to inform any appropriate changes to time frames, 
projects, and cost estimates for the remaining contract years; 

* ensure that testing of the electronic versions of the measures 
retooled by NQF that are being used or are planned for use in the 
Medicare and Medicaid EHR Incentive programs is completed in a timely 
manner to help identify potential errors and address issues of 
implementation; and: 

* develop a comprehensive plan that identifies the quality measurement 
needs of HHS programs and initiatives, including PPACA requirements, 
and provides a strategy for using the work NQF performs under the 
contract to help meet these needs. 

Agency and Other External Comments and Our Evaluation: 

We provided a draft of this report to HHS and NQF for review and 
comment. HHS neither agreed nor disagreed with our recommendations and 
provided general comments. NQF concurred with many of the findings in 
the report and provided clarification and additional context on the 
findings and recommendations. HHS and NQF's letters conveying their 
comments are reproduced in appendixes V and VI, respectively. In 
addition to the overall comments discussed below, we received 
technical comments from HHS and NQF, which we incorporated into our 
report as appropriate. 

HHS Comments: 

HHS's comments included separate general comments from CMS and ASPE 
that provided context on aspects of our findings and recommendations. 
CMS's comments stated that the draft report suggests that CMS must use 
all of the measures endorsed by NQF, and noted that not all NQF-
endorsed measures are suitable for HHS quality reporting and public 
reporting programs. Although our draft report did not state that CMS 
must use all of the measures endorsed by NQF, we modified it to note 
specifically, among other things, that all measures received under the 
contract may not be applicable to a particular HHS health care quality 
program or initiative. CMS also stated that the report suggests that 
CMS has not developed measurement plans for various provisions of 
PPACA related to quality reporting, public reporting, and value-based 
purchasing programs. CMS provided additional context for current 
planning efforts to address these requirements, including its Quality 
Measurement Task Force. The draft report acknowledged this and other 
CMS planning efforts to address the health care quality requirements 
contained in PPACA and noted that, as of August 2011, this initiative 
was just beginning. Further, while various efforts are underway and 
CMS's comments state that it has documented how quality measures will 
be used to address all relevant provisions of PPACA, CMS has not 
provided documentation of comprehensive plans to address PPACA 
requirements that include alignment across programs, detailed time 
frames to meet PPACA deadlines, or how it will use the NQF contract to 
help ensure that it receives the endorsed measures it needs to meet 
these requirements. 

ASPE's comments noted, with respect to our first recommendation, that 
HHS used all except two of the monitoring tools called for in the 
contract. As noted in the draft report, HHS began receiving the 
monthly financial graph--one of the two monitoring tools--from NQF in 
August 2011. Also, ASPE noted its plans to update its performance 
evaluation system with NQF performance information for the first 2 
contract years--the period January 14, 2009, through January 13, 2011--
and to complete a final performance evaluation at the end of the 
contract in January 2013, which is the end of the fourth contract 
year. It did not indicate any plans to conduct the annual performance 
evaluation for the third contract year--January 14, 2011, through 
January 13, 2012--which would be consistent with the contract's 
requirements. With respect to our second recommendation, ASPE provided 
technical comments and also told us that CMS issued a contract 
solicitation to test the retooled measures, but CMS did not receive 
any bids. Instead, ASPE noted in its comments that two of CMS's 
current contractors will conduct feasibility testing on 69 of the 113 
retooled measures that are planned for use in HHS's EHR Incentive 
programs. CMS does not plan to issue a solicitation for a new contract 
to test the feasibility of the remaining 44 retooled measures, which 
are currently being used in HHS's EHR Incentive Program. We noted 
these comments in the report. Regarding our third recommendation, ASPE 
stated that the measures that are not currently in "use" are being 
evaluated by HHS and that any conclusions that they will not be used 
are not accurate. Our draft report provided information on which 
measures were used or planned for use as of August 2011, and indicated 
that the remaining measures may be used in the future. Specifically, 
the report noted that HHS officials expect that many of these measures 
will be used in HHS programs or initiatives, and that HHS officials 
told us that they will review all the measures received under the 
contract to determine if they are applicable to their health care 
quality programs or initiatives. ASPE's comments also noted that our 
draft report did not include information on all NQF-endorsed measures 
used by the various agencies within HHS. As noted in the draft report, 
we relied on HHS to identify programs and initiatives across HHS that 
use or plan to use these health care quality measures and recognize 
that those included in our report may not represent a comprehensive 
list of all health care quality programs and initiatives. As we 
recommended in our report, having a comprehensive plan could help HHS 
identify programs or initiatives that use or plan to use health care 
quality measures, including those endorsed by NQF. 

NQF Comments: 

NQF's comments state that it is providing its services to HHS under a 
cost reimbursement contract, which is used in circumstances where 
aspects of performance, such as time frames, cost estimates, and scope 
of work, cannot be reasonably estimated, and therefore, should not be 
expected. As noted in the draft report, the contract type used for 
this work is used for efforts such as research, design, or study 
efforts where costs and technical uncertainties exist and it is 
desirable to retain as much flexibility as possible in order to 
accommodate change. However, the draft report also noted that this 
type of contract provides only a minimum incentive to the contractor 
to control costs. Given the risk associated with this type of 
contract, the fact that NQF has not met expected time frames on about 
half of its projects as of August 2011, and that NQF exceeded its 
initial cost estimates for some of its projects under its contract 
activities, it is especially important that HHS obtain detailed and 
timely information on NQF's performance and use that information to 
inform any appropriate changes to time frames, projects, and cost 
estimates for the remaining contract years, as noted in our 
recommendations. NQF's comments also state that time frames and costs 
for the work performed under the contract were initial estimates based 
on an early understanding of the work, that HHS and NQF understood 
that there would likely be changes to them as a result of the 
complexity and novelty of the work, and that they have worked 
collaboratively throughout the contract period to address these and 
other factors. As noted in the draft report, the final work plans, the 
technical proposal, and other documents that we reviewed included 
initial time frames for all projects and costs for the work performed 
during the contract year that were approved by HHS in collaboration 
with NQF. The draft report also notes several examples of reasons why 
the time frames and costs were modified over time. Contributing 
factors include the high volume of measures submitted, changes to the 
scope of work, and the novelty and complexity of the work. 

We are sending copies of this report to the Secretary of Health and 
Human Services and other interested parties. In addition, the report 
will be available at no charge on GAO's website at [hyperlink, 
http://www.gao.gov]. 

If you or your staff have any questions about this report, please 
contact me at (202) 512-7114 or at kohnl@gao.gov. Contact points for 
our Office of Congressional Relations and Office of Public Affairs can 
be found on the last page of this report. Other major contributors to 
this report are listed in appendix VII. 

Signed by: 

Linda T. Kohn: 
Director, Health Care: 

List of Committees: 

The Honorable Max Baucus: 
Chairman: 
The Honorable Orrin G. Hatch: 
Ranking Member: 
Committee on Finance: 
United States Senate: 

The Honorable Tom Harkin: 
Chairman: 
The Honorable Michael B. Enzi: 
Ranking Member: 
Committee on Health, Education, Labor, and Pensions: 
United States Senate: 

The Honorable Fred Upton: 
Chairman: 
The Honorable Henry A. Waxman: 
Ranking Member: 
Committee on Energy and Commerce: 
House of Representatives: 

The Honorable David Camp: 
Chairman: 
The Honorable Sander M. Levin: 
Ranking Member: 
Committee on Ways and Means: 
House of Representatives: 

[End of section] 

Appendix I: Health Care Quality Measurement Framework: 

Figure 5 illustrates a health care quality measurement framework of 
the various stages that a quality measure will go through, as 
described by the Department of Health and Human Services (HHS) and 
National Quality Forum (NQF) officials and others. These stages 
include measure development, endorsement, selection, and use, among 
others. This framework also shows examples of which entities, 
including HHS and NQF, are involved in each of the stages. 

Figure 5: Health Care Quality Measurement Framework: 

[Refer to PDF for image: illustration: continuous circle segments] 

1) HHS, NQF: Establishment of national priorities; 

2) HHS, NQF: Planning and identification of gap areas; 

3) HHS, Private Sector: Development of measures; 

4) NQF: Endorsement of measures; 

5) HHS, NQF: Development of data platforms[A]; 

6) HHS, NQF: Selection of measures for use; 

7) HHS, Private Sector: Use of measures; 

8) HHS, NQF: Maintenance and review of measures. 

Source: GAO analysis of NQF, HHS, and Congressional Research Service 
data. 

[A] NQF officials told us that the development of data platforms 
allows measure developers to create quality measures directly in an 
electronic format as well as retool--that is, convert measures from a 
paper-based format to an electronic format. For example, NQF's work to 
create a measure authoring tool provides measure developers with a 
tool to create standardized electronic measures that help capture 
information in electronic health records. NQF officials told us 
development of data platforms, such as through the use of the measure 
authoring tool, is increasingly important as NQF will begin to require 
that measure developers include electronic specifications in quality 
measures beginning in May 2012. 

[End of figure] 

HHS's Role in Planning and Identifying Gap Areas: 

As an example of actions taken during the second stage of this quality 
measurement framework, HHS officials described two different processes 
used for planning and identifying gap areas. 

The Centers for Medicare & Medicaid Services (CMS) Office of Clinical 
Standards and Quality[Footnote 67] has developed a standardized 
approach to identify quality measures that it uses in its health 
quality initiatives and programs using CMS's Measures Management 
System.[Footnote 68] The Measures Management System requires the 
convening of a technical expert panel in the initial planning stage. 
[Footnote 69] Once convened, the technical expert panel is expected to 
work with measure developers who will gather information that will 
help the panel determine whether measures need to be developed for a 
program or initiative. During this stage, measure developers may 
conduct environmental scans or literature reviews, to determine the 
existence of measures that could be used for a program or initiative. 
If a measure does not exist, then the developer will work with CMS to 
develop the needed measures for the program or initiative, including 
measure testing. Upon development of the measures, the technical 
expert panel will evaluate them based on (1) importance to making 
significant gains in health care quality and improving health 
outcomes, (2) scientific acceptability of the measure properties 
including tests of reliability and validity, (3) usability, and (4) 
feasibility. Measures recommended by the panel are generally submitted 
for NQF endorsement. 

In contrast, CMS's Center for Medicaid, Children's Health Insurance 
Program (CHIP), and Survey & Certification Office--the CMS center 
which implements CHIP--uses a measure identification process that 
relies on existing measures rather than development of new measures, 
according to officials. This office worked with a technical advisory 
group, the Subcommittee on Children's Healthcare Quality Measures for 
Medicaid and CHIP, to recommend an initial core set of measures for 
the CHIP. With assistance from CMS, the subcommittee evaluated 
measures based on importance, validity, and feasibility. CMS officials 
told us that they considered existing NQF-endorsed and non-NQF-
endorsed measures based on the measurement needs of the program, and 
relied on measure testing conducted by the measure developers. 
Officials stated that they have also relied on the subcommittee to 
evaluate candidate measures for Medicaid child health programs. 
Officials said that they are not required to submit measures that will 
be used for Medicaid programs for NQF endorsement. 

[End of section] 

Appendix II: How GAO Categorized NQF's 16 Tasks under the Contract 
into Nine Contract Activities: 

From January 14, 2010, through August 31, 2011, the National Quality 
Forum's (NQF) contract with the Department of Health and Human 
Services (HHS) included 16 tasks that NQF is required to perform. For 
purposes of our work, we categorized these tasks into nine contract 
activities. Specifically, in certain cases, we grouped activities that 
covered related areas of work into a single contract activity. For 
example, we consolidated the six administrative activities NQF is 
required to perform into a single contract activity. (See table 3 that 
shows how we consolidated these contract activities.) 

NQF was required to perform specific projects under the nine contract 
activities we identified. For example, under the endorsement contract 
activity, NQF was required to complete an endorsement project related 
to patient outcome measures. For purposes of our work, we identified 
and reviewed 63 projects NQF is required to perform under the nine 
contract activities, as shown in appendix III. 

Table 3: GAO's Categorization of the NQF Contract's 16 Tasks into Nine 
Activities: 

NQF contract's 16 tasks[A]: 
1. Opening meeting between HHS and NQF; 
2. Development of the annual work plan; 
3. Quality assurance (development of an internal evaluation plan and 
monthly call to discuss the plan); 
4. Weekly conference calls; 
5. Monthly progress reports; 
6. Develop public website for project documents[C]; 
Nine contract activities categorized by GAO: 
1. Administrative activity[B]. 

NQF contract's 16 tasks[A]: 
7. Formulation of national strategy and priorities for health care 
quality measures, as specified by the Medicare Improvements for 
Patients and Providers Act of 2008 (MIPPA); 
Nine contract activities categorized by GAO: 
2. Recommendations on a National Strategy and Priorities for Quality 
Measurement (MIPPA). 

NQF contract's 16 tasks[A]: 
8. Implementation of a consensus process for endorsement of health 
care quality measures (MIPPA); 
Nine contract activities categorized by GAO: 
3. Endorsement of Health Care Quality Measures (MIPPA). 

NQF contract's 16 tasks[A]: 
9. Implementation of a consensus process for maintenance of previously 
endorsed health care quality measures (MIPPA); 
Nine contract activities categorized by GAO: 
4. Maintenance of Endorsed Quality Measures (MIPPA). 

NQF contract's 16 tasks[A]: 
10. Promotion of electronic health records (MIPPA); 
Nine contract activities categorized by GAO: 
5. Promotion of the Development and Use of Electronic Health Records 
(MIPPA). 

NQF contract's 16 tasks[A]: 
11. Submission of annual report to Congress and the Secretary of 
Health and Human Services (MIPPA); 
12. Additional Requirements for Annual Report--Measure Gaps and 
Inadequacies and Input of the Multistakeholder Groups, as specified by 
the Patient Protection and Affordable Care Act (PPACA); 
Nine contract activities categorized by GAO: 
6. Annual Report to Congress and the Secretary of HHS (MIPPA and 
PPACA). 

NQF contract's 16 tasks[A]: 
13. Focused Measure Development, Harmonization, and Endorsement 
Efforts to Fill Critical Gaps in Performance Measurement (not 
specified by MIPPA or PPACA); 
Nine contract activities categorized by GAO: 
7. Other Health Care Quality Measurement Activity Not Identified in 
MIPPA or PPACA. 

NQF contract's 16 tasks[A]: 
14. Provision of Input on Priorities for the National Strategy for 
Quality Improvement in Healthcare (PPACA); 
15. Input to the Annual National Quality Strategy (PPACA); 
Nine contract activities categorized by GAO: 
8. Multistakeholder Input into HHS's National Strategy for Quality 
Improvement in Health Care (as required by PPACA). 

NQF contract's 16 tasks[A]: 
16. Selection of Quality Measures for Use in Payment Programs and 
Value-Based Purchasing Programs under PPACA, Other Private/Public 
Payers, and Other Programs (PPACA); 
Nine contract activities categorized by GAO: 
9. Multistakeholder Input on the Selection of Quality Measures for use 
in Payment Programs and Value-Based Purchasing Programs Under PPACA, 
Other Private/Public Payers, and Other Programs (as required by PPACA). 

Source: GAO analysis of MIPPA, PPACA, the NQF contract, the 2010 and 
2011 final MIPPA work plans, and the 2011 PPACA technical proposal. 

[A] The numbering of the 16 tasks, or contract activities, does not 
reflect the numbering used under the NQF contract with HHS. 

[B] A number of activities NQF performs under the HHS contract are 
administrative and are not directly related to NQF's work on health 
care quality measurement. Rather, these activities focus on project 
planning and contract management efforts. 

[C] Development of a public website was part of the 2010 final annual 
work plan but this work was canceled in 2011. 

[End of table] 

[End of section] 

Appendix III: Status of NQF's Nine Contract Activities and the 
Projects under Each Activity GAO Identified: 

The tables below provide a status update on the projects that the 
National Quality Forum (NQF) is required to complete under the nine 
contract activities we identified (see appendix II). The contract 
activities and the projects under the activities NQF is expected to 
perform are determined on an annual basis by the Department of Health 
and Human Services (HHS) and NQF. As a result, the number of projects 
under the contract activities varies by contract year. For our 
reporting period--January 2010 through August 2011--we determined that 
NQF was required to conduct work on 63 projects under the contract 
activities we reviewed. To determine initial time frames for each 
project, we calculated the approximate time between expected start and 
end dates established in NQF's 2009, 2010, and 2011 final annual 
Medicare Improvements for Patients and Providers Act of 2008 (MIPPA) 
work plans, the 2011 Patient Protection and Affordable Care Act 
(PPACA) technical proposal, and other NQF documents. Actual time 
frames were determined by calculating the approximate time between the 
actual start date and the actual date of completion. For projects that 
were not yet complete as of August 2011, we included an expected time 
frame based on the approximate difference between the actual start 
date and the expected date of completion. NQF and HHS officials stated 
that any changes to the initial time frames were approved by HHS. 

Table 4: Status of Projects under NQF's Contract Activity to Provide 
Recommendations on a National Strategy and Priorities for Quality 
Measurement, as of August 31, 2011: 

Projects under contract activity to provide recommendations on a 
national strategy and priorities for quality measurement (MIPPA): 
Formulate national strategies and priorities; 
Description of project: Report that prioritizes 20 high-impact 
Medicare conditions and associated measure gaps; 
Status: Completed; 
Expected time frame (estimated months): 14; 
Actual time frame (estimated months): 15. 

Projects under contract activity to provide recommendations on a 
national strategy and priorities for quality measurement (MIPPA): 
Evaluation of the uses of NQF-endorsed measures; 
Description of project: Report that investigates the use of NQF-
endorsed measures; 
Status: In progress; 
Expected time frame (estimated months): 9; 
Actual time frame (estimated months): expected 18. 

Projects under contract activity to provide recommendations on a 
national strategy and priorities for quality measurement (MIPPA): 
Measurement development and endorsement agenda; 
Description of project: Report that sets a national measure 
development and endorsement agenda; 
Status: Completed; 
Expected time frame (estimated months): 10; 
Actual time frame (estimated months): 10. 

Projects under contract activity to provide recommendations on a 
national strategy and priorities for quality measurement (MIPPA): 
Measurement framework for multiple chronic conditions; 
Description of project: Report that presents an analysis of measures 
being used to assess quality of care for people with multiple chronic 
conditions; 
Status: In progress; 
Expected time frame (estimated months): 14; 
Actual time frame (estimated months): expected 13[A]. 

Projects under contract activity to provide recommendations on a 
national strategy and priorities for quality measurement (MIPPA): 
Advice to the Office of the National Coordinator on meaningful use of 
measures; 
Description of project: Report that analyzes measures targeted for use 
in the Medicare Electronic Health Record Incentive Program, 
specifically examining how health information technology (IT) tools 
can improve the efficiency, quality, and safety of health care 
delivery; 
Status: Completed; 
Expected time frame (estimated months): 2; 
Actual time frame (estimated months): 2. 

Source: GAO analysis of NQF's contract with HHS, the 2010 and 2011 
final MIPPA annual work plans, and NQF's monthly progress and Internal 
Evaluation Plan (IEP) reports from January 2010 to August 2011. 

[A] The initial time line for this work was June 2010 through August 
2011, which is approximately 14 months. However, due to delays in 
approving the subcontractor NQF engaged to perform the work, the 
project was not started on time. As a result, the modified time line 
for the project is April 2011 through May 2012, which represents a 
delay of approximately 9 months. 

[End of table] 

Table 5: Status of Projects under NQF's Contract Activity to Endorse 
Health Care Quality Measures, as of August 31, 2011: 

Projects under NQF's contract activity to endorse health care quality 
measures (MIPPA)[A]: Patient outcomes; 
Description of project: Endorsement of health care outcome measures 
related to 20 high-impact Medicare conditions identified by the 
Centers for Medicare & Medicaid Services (CMS), and mental health, and 
child health conditions; 
Status: Completed; 
Expected time frame (estimated months): 14; 
Actual time frame (estimated months): 13-19[B]. 

Projects under NQF's contract activity to endorse health care quality 
measures (MIPPA)[A]: Patient safety; 
Description of project: Endorsement of patient safety measures and 
other projects related to patient safety, including a framework for 
public reporting of patient safety information, endorsement of serious 
reportable events (SRE), and collaboration with state-based reporting 
agencies; 
Status: In progress; 
Expected time frame (estimated months): 17; 
Actual time frame (estimated months): 23, expected 27[C]. 

Projects under NQF's contract activity to endorse health care quality 
measures (MIPPA)[A]: Nursing homes; 
Description of project: Endorsement of performance measures for 
chronic and postacute care nursing facilities; 
Status: Completed; 
Expected time frame (estimated months): 10; 
Actual time frame (estimated months): 17. 

Projects under NQF's contract activity to endorse health care quality 
measures (MIPPA)[A]: Evaluation of Consensus Development Process; 
Description of project: A comprehensive analysis of NQF's endorsement 
process by Mathematica Policy Research, Inc., to identify areas of 
improvement related to timeliness and effectiveness; 
Status: Completed; 
Expected time frame (estimated months): 9[D]; 
Actual time frame (estimated months): 18. 

Projects under NQF's contract activity to endorse health care quality 
measures (MIPPA)[A]: Child health quality; 
Description of project: Endorsement of measures that could be used in 
public reporting for certain conditions or cross-cutting areas 
applicable to the Medicaid population; 
Status: In progress; 
Expected time frame (estimated months): 11; 
Actual time frame (estimated months): expected 16[E]. 

Source: GAO analysis of NQF's contract with HHS, the 2010 and 2011 
final MIPPA annual work plans, and NQF's monthly progress and IEP 
reports from January 2010 to August 2011. 

[A] HHS officials told us that they consider an NQF measure endorsed 
when the board ratifies the measures. However, NQF is required to 
submit a final report to HHS following ratification to complete the 
project. For purposes of our report, we consider the project completed 
when the board endorses the measures. 

[B] The patient outcomes project was conducted in three phases. Phases 
I and II were initially expected to be completed in 13 months; 
however, the project was completed in 19 months. NQF officials told us 
a high volume of measures contributed to the delays. Phase II was 
initially expected to take 15 months but it took 17 months. 

[C] As of August 2011, NQF endorsed one patient safety measure under 
phase II of the endorsement project and expects to complete this phase 
in September 2011 following the review of an additional patient safety 
measure. Phase I is expected to be completed by December 2011. 

[D] In the 2009 final work plan, this project was expected to be 
completed between October 2009 and July 2010, or approximately 9 
months. NQF officials stated the delays in completing this work were a 
result of additional time needed for data collection efforts, among 
other things. 

[E] In August 2011, NQF announced endorsement of 41 out of 44 child 
health quality measures under the child health quality endorsement 
project. The remaining 3 measures are still under consideration and 
are scheduled to be reviewed for endorsement in September 2011. 

[End of table] 

Table 6: Status of Projects under NQF's Contract Activity to Maintain 
Previously Endorsed Health Care Quality Measures, as of August 31, 
2011: 

Projects under NQF's contract activity to maintain health care quality 
measures (MIPPA)[A]: Updated 3-year endorsement maintenance cycle 
review process projects (6 projects); 
Description of project: Six maintenance projects using NQF's process 
that reviews previously endorsed measures across 22 topic areas every 
3 years using three measure maintenance cycles.[B] The projects relate 
to the following conditions: 
* Cardiovascular; 
* Surgical; 
* Cancer; 
* Pulmonary/critical care; 
* Perinatal; 
* Renal care; 
Status: In progress; 
Expected time frame (estimated months): Various[C]; 
Actual time frame (estimated months): All ongoing. 

Projects under NQF's contract activity to maintain health care quality 
measures (MIPPA)[A]: 3-Year maintenance cycle review prior to updated 
process (3 projects); 
Description of project: Three maintenance projects conducted under the 
old NQF maintenance review process related to the following conditions: 
* Diabetes; 
* Mental health; 
* Musculoskeletal conditions; 
Status: Completed; 
Expected time frame (estimated months): No specific time frame[D]; 
Actual time frame (estimated months): 20. 

Projects under NQF's contract activity to maintain health care quality 
measures (MIPPA)[A]: Time-limited endorsement review; 
Description of project: Maintenance review of measures that have been 
granted a time-limited endorsement--measures that require additional 
testing to be conducted by measure developers within 12 months of 
endorsement; 
Status: In progress; 
Expected time frame (estimated months): No specific time frame[E]; 
Actual time frame (estimated months): No specific time frame. 

Projects under NQF's contract activity to maintain health care quality 
measures (MIPPA)[A]: Annual measure maintenance update; 
Description of project: Review of measures submitted by measure 
developers to determine whether the measure has undergone any change; 
Status: In progress; 
Expected time frame (estimated months): Various[F]; 
Actual time frame (estimated months): Ongoing. 

Projects under NQF's contract activity to maintain health care quality 
measures (MIPPA)[A]: Ad hoc review; 
Description of project: Review of measures that were initiated during 
the contract year by request and based on justifiable evidence to 
substantiate the review; 
Status: Completed; 
Expected time frame (estimated months): n/a; 
Actual time frame (estimated months): n/a[G]. 

Source: GAO analysis of NQF's contract with HHS, the 2010 and 2011 
final MIPPA annual work plans, and NQF's monthly progress and IEP 
reports from January 2010 to August 2011. 

Note: n/a = no time frame was established. 

[A] In addition to the 3-Year Maintenance Review cycles, NQF can also 
maintain measures through (1) annual updates, (2) ad hoc requests, and 
(3) time-limited endorsement reviews. As of August 2011, NQF has 
maintained 3 measures under ad hoc requests and 80 measures under time-
limited endorsement reviews. NQF officials told us they also began 
annual updates for 59 measures during our reporting period. 

[B] NQF has developed three cycles of review--cycles A, B, and C. 
Cycle A focuses on measures related to cardiovascular; surgery; 
prevention; endocrine; GU/GYN; mental health; and musculoskeletal 
disorders. Cycle B focuses on measures related to cancer; 
pulmonary/critical care; palliative and end-of-life care; Perinatal; 
renal; safety; disparities; and care coordination. Cycle C will focus 
on health care infrastructure; head/eye/ear/nose/throat conditions; 
infectious disease; neurology; patient experience and engagement; 
functional status; and gastrointestinal. NQF is expected to complete 
Cycle A by the end of 2010, Cycle B by the end of 2011, and Cycle C by 
the end of 2012. As of August 2011, the cardiovascular and surgery 
measures from Cycle A are still under review. 

[C] As of August 2011, NQF was conducting six maintenance projects 
under its updated 3-year review cycle. The initial and estimated time 
frames for these projects vary by project. For example, review for 
phases I and II of the cardiovascular measures is expected to take 15 
months and review for phase I of the surgical measures is expected to 
take 13 months and phase 2 is expected to take 16 months. NQF also 
initiated the review of renal measures in May 2011 and was initially 
expected to take 7 months; however, the time line has been extended to 
8 months. As of November 2011, NQF has issued a call for measures for 
Perinatal measures and expects to issue a call for cancer and 
pulmonary/critical care measures in November 2011 and January 2012, 
respectively. 

[D] NQF officials stated that no estimated time frame for completion 
was established for these three projects. Although officials stated 
they anticipated the length of the review to be similar to other NQF 
projects, they encountered delays that extended the time frame, such 
as a high volume of measures under review. 

[E] NQF officials provided us with a time line for completing time-
limited reviews that included an initial start date of January 2011 
but did not provide specific expected dates of completion. 

[F] NQF officials told us that the annual reviews did not begin until 
2011 and are conducted in cycles. 

[G] NQF officials told us that there is no time line established for 
ad hoc reviews because requests for ad hoc reviews will be considered 
by NQF on a case-by-case basis and can be made at any time. 

[End of table] 

Table 7: Status of Projects under NQF's Contract Activity to Promote 
the Development and Use of Electronic Health Records, as of August 31, 
2011: 

Project under NQF's contract activity to promote the development and 
use of electronic health records (MIPPA): Measure authoring tool; 
Description of project: Development of a software tool that measure 
developers will use to create standardized eMeasures; 
Status: In progress; 
Expected time frame (estimated months): 15; 
Actual time frame (estimated months): expected 28. 

Project under NQF's contract activity to promote the development and 
use of electronic health records (MIPPA): Clinical decision support; 
Description of project: Report that provides a framework for Clinical 
Decision Support rules and key elements for health IT systems to help 
ensure compatibility and functionality with the Quality Data Model; 
Status: Completed; 
Expected time frame (estimated months): 14; 
Actual time frame (estimated months): 17. 

Project under NQF's contract activity to promote the development and 
use of electronic health records (MIPPA): eMeasure creation and 
testing; 
Description of project: Establish a uniform standard for electronic 
measures; 
Status: Completed; 
Expected time frame (estimated months): 13; 
Actual time frame (estimated months): 13. 

Project under NQF's contract activity to promote the development and 
use of electronic health records (MIPPA): Quality data model; 
Description of project: Maintenance and revision of a health IT model 
that allows data from electronic health records to be collected and 
used in quality measurement; 
Status: In progress; 
Expected time frame (estimated months): 28; 
Actual time frame (estimated months): expected 28[A]. 

Project under NQF's contract activity to promote the development and 
use of electronic health records (MIPPA): Health information 
technology (IT) utilization expert panel; 
Description of project: Expert panel convened to examine the 
information needed to measure effective health IT use; 
Status: Completed; 
Expected time frame (estimated months): 12; 
Actual time frame (estimated months): 14. 

Project under NQF's contract activity to promote the development and 
use of electronic health records (MIPPA): Initial measure retooling; 
Description of project: "Retooling"--that is, converting 113 NQF-
endorsed health care quality measures from paper-based format to an 
eMeasure format; 
Status: Completed; 
Expected time frame (estimated months): 19; 
Actual time frame (estimated months): 22. 

Project under NQF's contract activity to promote the development and 
use of electronic health records (MIPPA): Update of retooled measures; 
Description of project: Publish a list of updated versions of the 
initial 113 measures under the contract that describes any changes 
made to the measures, including those identified during public comment 
and by a review panel; 
Status: In progress; 
Expected time frame (estimated months): 10; 
Actual time frame (estimated months): expected 10. 

Project under NQF's contract activity to promote the development and 
use of electronic health records (MIPPA): eMeasure syntactical review 
panel; 
Description of project: Expert panel that conducts a thorough review 
of the retooled measures to help ensure that a measure's intent 
remains intact for continued NQF endorsement; 
Status: In progress; 
Expected time frame (estimated months): 9; 
Actual time frame (estimated months): 15, expected 19[B]. 

Source: GAO analysis of NQF's contract with HHS, the 2010 and 2011 
final MIPPA annual work plans, and NQF's monthly progress and IEP 
reports from January 2010 to August 2011. 

[A] NQF officials told us they anticipate a number of versions will be 
published annually under the contract to allow for ongoing review and 
updates. NQF expects to publish an updated version for public use in 
January 2012. 

[B] In June 2011, the review panel completed its review of the 113 
retooled measures and expects to publish a report that synthesizes 
comments on and updates to the measures by October 2011. NQF officials 
told us that a request by HHS to expand the scope of this work 
extended the time line by 3 months. 

[End of table] 

Table 8: Status of Projects under NQF's Contract Activity to Submit an 
Annual Report to Congress and the Secretary of HHS, as of August 31, 
2011: 

Project under NQF's contract activity to submit an annual report to 
Congress and the Secretary of HHS (MIPPA and PPACA): Report due March 
1, 2010; 
Description of project: Summary of NQF's progress on activities under 
the contract during the first contract year (2009); 
Status: Completed; 
Expected time frame (estimated months): 12; 
Actual time frame (estimated months): 12. 

Project under NQF's contract activity to submit an annual report to 
Congress and the Secretary of HHS (MIPPA and PPACA): Report due March 
1, 2011; 
Description of project: Summary of NQF's progress on activities under 
the contract during the second contract year (2010); 
Status: Completed; 
Expected time frame (estimated months): 13; 
Actual time frame (estimated months): 12. 

Project under NQF's contract activity to submit an annual report to 
Congress and the Secretary of HHS (MIPPA and PPACA): Report due March 
1, 2012; 
Description of project: Summary of NQF's progress on activities under 
the contract, including a report on gaps in endorsed and nonendorsed 
health care quality measures and a summary of activities conducted by 
multistakeholder groups during the third contract year (2011); 
Status: In progress; 
Expected time frame (estimated months): 13; 
Actual time frame (estimated months): expected 12. 

Source: GAO analysis of NQF's contract with HHS, the 2010 and 2011 
final MIPPA annual work plans, and NQF's monthly progress and IEP 
reports from January 2010 to August 2011. 

[End of table] 

Table 9: Status of Projects under NQF's Contract Activity to Provide 
Multistakeholder Input into HHS's National Strategy for Quality 
Improvement in Healthcare, as of August 31, 2011: 

Project under NQF's contract activity to provide multistakeholder 
input to HHS's National Quality Strategy (PPACA): Recommendations on 
HHS's National Strategy for Quality Improvement in Healthcare (2010); 
Description of project: Multistakeholder input on national priorities 
related to improvement in population health and in delivery of health 
care services that HHS will consider when developing the National 
Quality Strategy issued in 2011; 
Status: Completed; 
Expected time frame (estimated months): 1; 
Actual time frame (estimated months): 1. 

Project under NQF's contract activity to provide multistakeholder 
input to HHS's National Quality Strategy (PPACA): Provision of input 
on priorities for HHS's annual National Quality Strategy; 
Description of project: Multistakeholder input on a plan for measuring 
and improving health and health care with a focus on the six 
priorities identified in HHS's National Quality Strategy; 
Status: In progress; 
Expected time frame (estimated months): 5; 
Actual time frame (estimated months): expected 6. 

Project under NQF's contract activity to provide multistakeholder 
input to HHS's National Quality Strategy (PPACA): Provision of input 
on Partnership for Patients[A]; 
Description of project: Quarterly webinars, facilitated by the 
National Priorities Partnership (NPP), for purposes of gathering input 
issues related to HHS's Partnership for Patients initiative; 
Status: In progress; 
Expected time frame (estimated months): 7; 
Actual time frame (estimated months): expected 5. 

Project under NQF's contract activity to provide multistakeholder 
input to HHS's National Quality Strategy (PPACA): Care coordination; 
Description of project: A two-phase project that will include an 
endorsement project related to prioritized gap areas related to care 
coordination, such as transitions across settings and providers; 
and (2) two reports that will address issues related to care 
coordination measures, such as readiness of health information 
technology systems; 
Status: In progress; 
Expected time frame (estimated months): 
Phase 1: 6; 
Phase 2: 13; 
Actual time frame (estimated months): 
Phase 1: expected 10; 
Phase 2: expected 15. 

Project under NQF's contract activity to provide multistakeholder 
input to HHS's National Quality Strategy (PPACA): Palliative and end-
of-life care; 
Description of project: Endorsement project for individual and 
composite measures or cross-cutting measures related to palliative and 
end-of life care; 
Status: In progress; 
Expected time frame (estimated months): 9; 
Actual time frame (estimated months): expected 9. 

Project under NQF's contract activity to provide multistakeholder 
input to HHS's National Quality Strategy (PPACA): Patient safety; 
Description of project: Initiative that includes (a) an endorsement 
project for patient safety complications measures; (b) work related to 
SRE codification; and (c) development of an integrated strategy for 
high-profile areas in patient safety; 
Status: In progress; 
Expected time frame (estimated months): 
(a) 9 or 10, to be determined[B]; 
(b) 7; 
(c) 6; 
Actual time frame (estimated months): (a) expected 9; 
(b) expected 8; 
(c) expected 6. 

Project under NQF's contract activity to provide multistakeholder 
input to HHS's National Quality Strategy (PPACA): Population health; 
Description of project: Endorsement project for (a) preventive 
services and population health measures and (b) a report that provides 
information related to key topic areas related to population health, 
including applicable NQF evaluation criteria; 
Status: In progress; 
Expected time frame (estimated months): 
(a) 9; 
(b) 5; 
Actual time frame (estimated months): 
(a) expected 9; 
(b) expected 6. 

Project under NQF's contract activity to provide multistakeholder 
input to HHS's National Quality Strategy (PPACA): Patient reported 
outcomes; 
Description of project: Two reports--one that addresses issues related 
to use of patient reported outcomes measures, including methodological 
issues, and another that discusses information needed to prepare 
measures for the endorsement process; 
Status: In progress; 
Expected time frame (estimated months): 9; 
Actual time frame (estimated months): expected 9. 

Source: GAO analysis of NQF's contract with HHS, the 2011 PPACA 
technical proposal, and NQF's monthly progress and IEP reports from 
January 2010 to August 2011. 

[A] In April 2011, HHS launched the Partnership for Patients, which is 
a national partnership with the goal to save 60,000 lives by improving 
patient safety through reduction in preventable injuries, 
complications, and hospital readmissions over the next 3 years. 

[B] NQF officials told us that they expect the time lines for the 
endorsement project to be extended because the project will be 
conducted in phases to accommodate additional measures. However, as of 
August 2011, NQF officials told us they are still finalizing the scope 
of work for this project, including determining if there will be any 
impact on the overall time line. 

[End of table] 

Table 10: Status of Projects under NQF's Contract Activity to Provide 
Multistakeholder Input on Selection of Quality Measures, as of August 
31, 2011: 

Project under NQF's contract activity to provide multistakeholder 
input on selection of quality measures (PPACA): Multistakeholder 
prerulemaking input; 
Description of project: Multistakeholder input provided annually by 
the Measure Applications Partnership (MAP) to HHS on the selection of 
measures to be implemented through the federal rulemaking process for 
various programs[A]; 
Status: In progress; 
Expected time frame (estimated months): 11; 
Actual time frame (estimated months): expected 11. 

Project under NQF's contract activity to provide multistakeholder 
input on selection of quality measures (PPACA): Measures for use in 
improvement of quality reporting programs; 
Description of project: Multistakeholder input provided to HHS by the 
MAP to help develop a coordination strategy for performance 
measurement and identify health care quality measures for use in four 
quality reporting programs: (a) physicians, (b) postacute care 
programs under Medicare, (c) PPS-exempt cancer hospitals, and (d) 
hospice care; 
Status: In progress; 
Expected time frame (estimated months): 
(a) 7; (b) 11; (c) 15; (d) 15; 
Actual time frame (estimated months): (a) expected 7; (b) expected 11; 
(c) expected 15; (d) expected 15. 

Project under NQF's contract activity to provide multistakeholder 
input on selection of quality measures (PPACA): Measures for dual 
eligible beneficiaries; 
Description of project: Multistakeholder input provided to HHS by the 
MAP on a measurement strategy and potential new health care quality 
measures that address quality issues identified for dual eligible 
beneficiaries, such as multiple chronic conditions; 
Status: In progress; 
Expected time frame (estimated months): 15; 
Actual time frame (estimated months): expected 15. 

Project under NQF's contract activity to provide multistakeholder 
input on selection of quality measures (PPACA): Measurement strategy 
for readmissions and healthcare-acquired conditions (HAC); 
Description of project: Multistakeholder input provided to HHS by the 
MAP on a coordination strategy for readmission and HAC measurement 
across public and private payers; 
Status: In progress; 
Expected time frame (estimated months): 18; 
Actual time frame (estimated months): expected 17. 

Source: GAO analysis of NQF's contract with HHS, the 2011 PPACA 
technical proposal, and NQF's monthly progress and IEP reports from 
January 2010 to August 2011. 

[A] The MAP will provide input on measures beginning February 1, 2012, 
based on a list of measures published on December 1 of each year by 
the Secretary of HHS. The 11 programs included in this prerulemaking 
input are: hospice, hospital inpatient, hospital outpatient, 
physicians' offices, cancer hospitals, end-stage renal disease (ESRD) 
facilities, inpatient rehabilitation facilities, long-term care 
hospitals, CMS hospital value-based purchasing, psychiatric hospitals, 
and home health care. 

[End of table] 

Table 11: Status of Projects under NQF's Other Health Care Quality 
Measurement Activity Not Identified in MIPPA or PPACA, as of August 
31, 2011: 

Project under NQF's other health care quality measurement contract 
activity: Imaging efficiency measures; 
Description of project: Endorsement project for imaging efficiency 
measures because Medicare spends approximately $14 billion annually on 
outpatient imaging studies; 
Status: Completed; 
Expected time frame (estimated months): 8; 
Actual time frame (estimated months): 15-18[A]. 

Project under NQF's other health care quality measurement contract 
activity: Resource-use measures; 
Description of project: Endorsement project for measures to help 
assess the cost of health care services provided; 
Status: In progress; 
Expected time frame (estimated months): 12-14[B]; 
Actual time frame (estimated months): expected 11-14. 

Project under NQF's other health care quality measurement contract 
activity: Resource-use white paper; 
Description of project: White paper on resource-use measures that will 
inform a 2011 endorsement project on this topic; 
Status: In progress; 
Expected time frame (estimated months): 9; 
Actual time frame (estimated months): expected 22. 

Project under NQF's other health care quality measurement contract 
activity: Geographic-level efficiency measurement white paper; 
Description of project: White paper on geographic-level efficiency 
measures that are used for quality measure reporting at the geographic 
and population level; 
Status: Canceled; 
Expected time frame (estimated months): canceled (June 2010). 

Project under NQF's other health care quality measurement contract 
activity: Harmonization; 
Description of project: Report that provides operational guidance 
related to harmonization to be used in future NQF consensus 
development projects; 
Status: Complete; 
Expected time frame (estimated months): 9; 
Actual time frame (estimated months): 9[C]. 

Project under NQF's other health care quality measurement contract 
activity: ICD-10 conversion[D]; 
Description of project: Report that examines implications of the 
transition from an HHS code set that is used to classify health care 
on NQF's measure maintenance process and provides recommendations to 
assist measure developers and NQF during the transition; 
Status: Complete; 
Expected time frame (estimated months): 11; 
Actual time frame (estimated months): 15. 

Project under NQF's other health care quality measurement contract 
activity: Measure development[E]; 
Description of project: Maintenance of subcontracts with eight measure 
developers to allow for development of critical measures as needed; 
Status: Not started; 
Expected time frame (estimated months): n/a; 
Actual time frame (estimated months): [Empty]. 

Project under NQF's other health care quality measurement contract 
activity: Regionalized emergency medical care services; 
Description of project: Identification of existing quality measures 
and gap areas related to measurement of regionalized emergency care 
services to inform creation of a framework to guide measure 
development in this area; 
Status: In progress; 
Expected time frame (estimated months): 14; 
Actual time frame (estimated months): expected 18. 

Source: GAO analysis of NQF's contract with HHS, the 2010 and 2011 
final MIPPA annual work plans, and NQF's monthly progress and IEP 
reports from January 2010 to August 2011. 

Note: n/a = no time frame was established: 

[A] The imaging efficiency measures endorsement project was initially 
expected to endorse measures by July 2010; however, due to measure 
harmonization efforts, NQF divided the endorsement review in two 
phases. The first six measures were endorsed in February 2011. In May 
2011, NQF issued a second report announcing endorsement of another 
measure. 

[B] The resource use endorsement project will be conducted in two 
cycles--cycle 1 is expected to be completed in 11 months and cycle 2 
is expected to be completed in 14 months. 

[C] The initial time line for completing this project was February 
2010 through November 2010; however, the actual time line for 
completion was from March 2010 through December 2010, which delayed 
delivery of the report by 1 month. 

[D] HHS uses the International Classification of Diseases, Ninth 
Revision, Clinical Modification (ICD-9-CM) code sets to classify 
health care. In 2013, HHS will transition to the Tenth Revision, or 
ICD-10-CM/PCS. The ICD-10 update activity aims to help HHS prepare for 
this transition by examining the impact of the change on measure 
development and submitting measures for NQF endorsement. 

[E] HHS officials stated that HHS has never implemented this project 
and likely will not do so through the remainder of the contract period. 

[End of table] 

Table 12: Status of Projects under NQF's Administrative Contract 
Activity: 

Project under NQF's administrative contract activity: Opening meeting 
with HHS; 
Description of project: Meeting between NQF and HHS officials at the 
beginning of the contract year to review activities to be performed 
during for the year; 
Status: Completed; 
Expected time frame (estimated months): This is a single event held 
annually. 

Project under NQF's administrative contract activity: 2010 annual work 
plan; 
Description of project: Development of a draft and final work plan 
that outlines specific activities to be performed during the 2010 
contract year; 
Status: Completed; 
Expected time frame (estimated months): 3; 
Actual time frame (estimated months): 4. 

Project under NQF's administrative contract activity: 2011 annual work 
plan; 
Description of project: Development of a draft and final work plan 
that outlines specific activities to be performed during the 2011 
contract year related to NQF's activities funded by MIPPA; 
Status: Completed; 
Expected time frame (estimated months): 5; 
Actual time frame (estimated months): 6. 

Project under NQF's administrative contract activity: Quality 
assurance and monthly Internal Evaluation Plan (IEP) reports; 
Description of project: Monthly report compiled by NQF that includes 
process and outcome measures to help monitor NQF's progress under the 
contract; 
Status: Completed; 
Expected time frame (estimated months): All completed in reporting 
period, ongoing annually. 

Project under NQF's administrative contract activity: Weekly 
conference call with HHS; 
Description of project: Weekly conference call with HHS and NQF to go 
over projects under each activity; 
Status: Completed; 
Expected time frame (estimated months): Not all held in reporting 
period,[A] ongoing annually. 

Project under NQF's administrative contract activity: Monthly progress 
report; 
Description of project: Monthly report compiled by NQF that includes a 
summary for each activity and the status of project planning, 
implementation, management, quality assurance, and evaluation that has 
occurred during the previous month's contract period; 
Status: Completed; 
Expected time frame (estimated months): All completed in reporting 
period, ongoing annually[B]. 

Project under NQF's administrative contract activity: Public website; 
Description of project: Overhaul and maintenance of NQF's website to 
allow measure developers, members, and the public easier access to 
relevant documents; 
Status: Canceled; 
Expected time frame (estimated months): canceled (February 2011). 

Source: GAO analysis of NQF's contract with HHS, the 2010 and 2011 
final MIPPA annual work plans, and NQF's monthly progress and IEP 
reports from January 2010 to August 2011. 

[A] NQF officials told us that this meeting did not occur each week, 
as scheduled, because in some instances those involved determined it 
was unnecessary. 

[B] From January 2010 to June 2010, NQF was required to submit its 
monthly progress reports by the 1st of every month; however, it did 
not submit these reports until the 15th of the month. In June 2010, 
HHS modified the due date to the 15th, beginning in July 2010. Since 
that time, all reports have been submitted on time. 

[End of table] 

[End of section] 

Appendix IV: Summary of Projects to Review the National Quality 
Forum's Endorsement Process: 

As part of a project under its contract with the Department of Health 
and Human Services (HHS), NQF was required to review its endorsement 
process. To complete this project, the National Quality Forum (NQF) 
subcontracted with Mathematica Policy Research, Inc. (Mathematica), to 
conduct a review of NQF's endorsement process, as requested by HHS. 
HHS officials stated that, given the importance of the endorsement 
process as part of the health care quality measurement framework, they 
requested that an objective and thorough review of NQF's endorsement 
process that focused on timeliness, efficiency, and effectiveness 
should be conducted. For example, they stated that they were 
interested in whether there were any efficiencies that could be 
implemented to shorten the process while maintaining an objective 
review of the health care quality measures that were evaluated under 
the process. Mathematica initiated its review of NQF's endorsement 
process in October 2009 and completed the work in December 2010. 

In December 2010, Mathematica submitted a final report to NQF and 
recommended eight areas where improvements could be made and 
inefficiencies could be addressed in the endorsement process. In the 
final report, Mathematica noted that the current process is lengthy 
and the timeliness of the endorsement projects varies substantially. 
The report further noted that the length of the endorsement process 
affects the availability of endorsed measures for end users, such as 
HHS. To help reduce the time required to complete projects, 
Mathematica recommended that NQF create a schedule for its endorsement 
process for measure developers and develop feasible time lines that 
include clear goals for each endorsement project. 

As of May 2011, NQF officials stated that NQF has taken steps or plans 
to take steps in its future projects to address the eight areas for 
improvement Mathematica identified. For example, as of May 2011, NQF 
has solicited measures earlier based on a tentative annual project 
schedule to reduce the time lines of its endorsement process and 
reduced the period for voting by NQF member organizations from 30 to 
15 days. NQF officials stated that they believe their efforts to 
implement the recommendations will shorten the time lines for the 
endorsement projects by 3 to 4 months without compromising the 
integrity of the endorsement process and measures to be evaluated 
under the process. HHS officials stated Mathematica's recommendations 
were valuable because much of the work under the NQF contract needs to 
be completed in an accelerated time line to help fill critical 
measurement gaps associated with HHS's health care quality programs 
and initiatives. They noted that it is too soon to tell the effects of 
these changes on the endorsement process, but they plan to monitor 
implementation of the changes in NQF's 2011 endorsement projects under 
the contract. In addition, as of September 2011, HHS approved a new 
project under the contract to identify how the endorsement process can 
best align with HHS's time frame for needed measures. As part of this 
project, NQF is expected to work with a consulting group to identify 
key performance metrics and define milestones and time lines to help 
streamline its endorsement process. 

[End of section] 

Appendix V: Comments from the Department of Health and Human Services: 

Department of Health & Human Services: 
Office of The Secretary:
Assistant Secretary for Legislation: 
Washington, DC 20201: 

December 16, 2011: 

Linda Kohn: 
Director, Health Care: 
U.S. Government Accountability Office: 
441 G Street NW: 
Washington, DC 20548: 

Dear Ms. Kohn: 

Attached are comments on the U.S. Government Accountability Office's 
(GAO) draft report entitled, "Health Care Quality Measurement: HHS 
Should Address Contractor Performance and Plan for Needed Measures" 
(GAO 12-136). 

The Department appreciates the opportunity to review this report 
before its publication. 

Sincerely, 

Signed by: 

Jim R. Esquea: 
Assistant Secretary for Legislation: 

Attachment: 

[End of letter] 

General Comments Of The Department Of Health And Human Services (HHS) 
On The Government Accountability Office's (GAO) Draft Report Entitled, 
"Health Care Quality Measurement: HHS Should Address Contractor 
Performance And Plan For Needed Measures" (GAO-12-136): 

The Department appreciates the opportunity to review and comment on 
this draft report. 

Centers for Medicare & Medicaid Services (CMS) Response: 

The draft report suggests that CMS must use all of the measures 
endorsed by NQF in the various maintenance and Consensus Development 
Process projects. However, CMS, when considering NQF endorsed measures 
for implementation, must also consider program requirements, 
administrative burden, suitability for program implementation, and the 
resources for data collection. During an NQF open Call for Measures, 
any measure developer can submit a measure for endorsement. Each 
developer has their purpose for development and plans for 
implementation, which may or may not involve HHS quality reporting and 
public reporting programs. The NQF MAP process will help identify 
measures that are suitable and pertinent for CMS quality reporting and 
public reporting programs. 

The report also suggests CMS has not developed measure development 
plans for the various Affordable Care Act (ACA) sections involving 
quality reporting, public reporting, and value-based purchasing 
programs. We would like to note that the CMS Quality Measures Task 
Force has identified measure priorities, current measures implemented, 
and measures under consideration for future implementation for 23 
quality programs. The measures will be reviewed and analyzed by the 
NQF Measure Applications Partnership (MAP) for multi-stakeholder input 
(Affordable Care Act section 3014). These measures will be considered 
for selection in the 2012 Federal Rulemaking process and may be used 
for future program implementation. During the 2012 Federal Rulemaking 
process, measures for programs may be finalized for future years and 
payment determinations. 

CMS has analyzed program and measurement needs and has developed plans 
for each of the ACA sections involving quality reporting and public 
reporting. For instance, ACA has established 5 new quality reporting 
programs for Inpatient Rehabilitation Facilities (IRFs), Long-Term 
Care Hospitals (LTCHs), Hospices, PPS-exempt Cancer Hospitals, and 
Psychiatric Hospitals. We would like to clarify where we are with 
these 5 programs: Three of these programs (IRE, LTCH, and Hospice) 
finalized the first set of measures for the 2014 payment year through 
the Federal Rulemaking process. Measures have been identified for 
future program implementation for PPS-exempt Cancer Hospitals and 
Psychiatric Hospitals, which will he proposed through the Federal 
Rulemaking process in 2012. CMS continues to track the progress of ACA 
measure development and implementation activities. 

Finally, CMS has undertaken the following processes to identify the 
initial core set and address measurement development needs within 
Medicaid and the Children's Health Insurance Program (CHIP): 

The Children's Health Insurance Reauthorization Act of 2009 (CHIPRA, 
P.L. 111-3) directed the Secretary of IIHS to identify an initial core 
set of health care quality measures for children in Medicaid/CHIP 
based on measures currently in use. In 2009, under delegated authority 
of the Secretary, CMS began collaborating with the Agency for 
Healthcare Research and Quality (AHRQ) to identify the initial core 
set. In order to include a wide range of national expertise in 
identifying measurement priority areas and the initial core set of 
measures, AHRQ established a Subcommittee of its National Advisory 
Council (SNAC). The Subcommittee reviewed and evaluated measures from 
nationally recognized sources, including measures endorsed by the 
National Quality Forum (NQF). A key priority used in selecting the 
initial core set measures was whether or not the measure was relevant 
to the Medicaid/CHIP population. While NQF endorsement signifies that 
measures have been deemed as meeting certain criteria for scientific 
soundness, validity and reliability, requiring NQF endorsement would 
have eliminated inclusion of measures in the initial core set that are 
relevant for assessing important aspects of care for the Medicaid/CHIP 
population. In addition to working with AHRQ's Subcommittee of health 
care experts, CMS also held a listening session to provide State 
Medicaid and CHIP programs the opportunity to comment on the draft 
initial core set of quality measures. Additionally, a Federal Quality 
Work Group comprised of representatives from various Federal agencies 
reviewed the measures. 

Supplementing the initial core measures is the development of the 
CHIPRA Pediatric Quality Measures Program (PQMP). Working in 
partnership, AHRQ and CMS awarded grants to seven Centers of 
Excellence in Pediatric Quality Measures in March 2011, which comprise 
the PQMP. These Centers of Excellence are charged with refining the 
initial core set of measures to make them more broadly applicable 
across types of payers and developing additional quality measures that 
address dimensions of care, where standardized measures do not 
currently exist. 

Section 2701 of the Affordable Care Act requires the development of a 
core set of health quality measures for adults eligible for benefits 
under Medicaid. The statute parallels the requirement under section 
1139A of the Act to identify and publish a recommended initial core 
set of quality measures for children in Medicaid and the Children's 
Health Insurance Program. CMS used a similar process to identify the 
initial set of health care quality measures for Medicaid-eligible 
adults and partnered with AHRQ to collaborate on the identification of 
the initial core set of health care quality measures for adults. 
Working through its National Advisory Council for Healthcare Research 
and Quality which provides advice and recommendations to the Director 
of AHRQ and to the Secretary of HHS on priorities for a national 
health services research agenda, AHRQ created a Subcommittee in the 
fall of 2010 to evaluate candidate measures for the initial core set. 
The Subcommittee consisted of State Medicaid representatives, health 
care quality experts, and representatives of health professional 
organizations and associations, and was charged with considering the 
health care quality needs of adults (ages 18 and older) enrolled in 
Medicaid in its recommendation for an initial core set of measures to 
HHS. The draft list of 51 measures were posted for public comment 
through a Federal Register (75 FR 82397) notice from December 30, 2010 
to March 1, 2011. 

In August 2011, CMS and AI IRQ reconvened the Subcommittee to the 
National Advisory Council to review the public comments, identify 
measures that ensured comprehensive representation of variables 
affecting Medicaid-eligible adults, and consider ways to decrease the 
number of measures in the set. Five criteria were used to evaluate 
measures: importance; scientific evidence supporting the measure; 
scientific soundness of the measure; current use in and alignment with 
existing Federal programs; and feasibility of State reporting.
The initial core set of quality measures for voluntary annual 
reporting by States, to be published by January 1, 2012, was 
determined based on recommendations from the AHRQ Subcommittee to the 
National Advisory Council for Healthcare Research and Quality, as well 
as public comments, before being finalized by the Secretary. These 
core set measures will support HHS and its State partners in 
developing a quality-driven, evidence-based, national system for 
measuring the quality of health care provided to Medicaid-eligible 
adults. 

Office of the Assistant Secretary for Evaluation and Planning (ASPE) 
Response: 

The National Quality Forum (NQF) contract was established to meet two 
broad objects outlined in the Medicare Improvement for Patients and 
Providers Act (MIPPA) and expanded by the Patient Protection and 
Affordable Care Act (ACA). The first was to establish and implement a 
process to endorse measures of health care performance as a national 
quality standard for HHS programs. This objective is typical of other 
work conducted under Federal contract. The second was to develop and 
implement this process under a contract with a 'consensus-based' 
entity with expertise in joining public and private stakeholders in 
decision-making, and to build a consensus to address the current 
disparate quality improvement efforts across the Nation. This 
nationwide public-private effort is designed to address national 
health care priorities and ultimately be the best strategy for quality 
improvement in Federally-sponsored programs. The implications of the 
second, less typical objective, is key to understanding the management 
flexibility required of this contract. 

Among the innovative advances toward improving health care under the 
contract, NQF created the National Priorities Partnership of 48 
leading organizations which provided input on national priorities for 
improvement in population health and delivery of health care services. 
NQF also undertook a harmonization project to address duplication and 
overlapping of measures which result from separate quality initiatives 
that focus on different settings and patient populations; it convened 
a stakeholder Steering Committee to develop and publish operational 
guidance for achieving global harmonization in all settings within 
future NQF consensus development projects. The final report, Guidance 
for Measure Harmonization, was completed in January 2011. A pragmatic 
and utilitarian example of the multi-stakeholder consensus process 
undertaken by the NQF is the selection of quality measures for payment 
programs and value-based purchasing. The challenges of addressing the 
complex, innovative and ultimately practical projects, required by 
law, are being successfully met through the multiple perspectives that 
have been facilitated by the flexible management of the contract. 

The first issue for which GAO makes a recommendation is: "HHS did not 
use all monitoring tools required under the NQF contract." With two 
exceptions, HHS used all of the monitoring tools called for in the 
contract, and developed an additional tool as well. These tools 
include: an Internal Evaluation Plan (IEP) and monthly IEP calls 
between NQF and the Project Officer; weekly conference calls to review 
current activities; and monthly progress reports. In addition, NQF 
produced the Annual Reports to Congress, HHS conducted the Secretarial 
review of the Report and both were published in the Federal Register 
as required by law within the mandated timeframes. Finally, beginning 
January 2010, ASPE produced a monthly Gantt chart which depicts the 
progress of each of the more than 60 tasks required by MIPPA and ACA 
against its identified timeline. As GAO notes, HHS did not use two of 
the required monitoring tools: a financial graph and the annual 
performance evaluation. As acknowledged in the Report, HHS began 
receiving the monthly financial graph in August 2011. The financial 
graph plots the actual vs. planned direct costs on each of the active 
projects. 

Effective October 1, 2010, the National Institutes of Health (NIH) 
Contractor Performance System (CPS) was replaced with the Department 
of Defense's Contractor Performance Assessment Reporting System 
(CPARS) for documenting the evaluation of contractor performance. CMS 
experienced issues transitioning from the NH-I system to the CPARS 
system. CMS is currently able to access the CPARS system and plans to 
update the CPARS system with the NQF performance information for the 
period January 14, 2009 through January 13, 2011. CMS continually 
monitors contractor performance through the receipt and review of 
monthly deliverables including monthly status reports. A final 
performance evaluation will be completed in the CPARS system at the 
end of the contract in January 2013. 

The final recommendation is to develop a comprehensive plan for HHS 
quality measure needs. In the GAO report, 'use' of quality measures 
and 'plans for use' of quality measures are defined as either being 
published as a final rule or in a notice of proposed rule-making 
(NPRM) or other NHS documentation. The measures that are not yet in 
"use" are being evaluated by HHS, so any conclusions that they will 
not be used are not accurate. For example, not all HHS Operating 
Divisions were involved in the GAO's study; thus, their use or plan 
for use of endorsed quality measures were not included in this report. 
However, IHS, HRSA, and SAMHSA all include NQF-endorsed measures in 
their quality measurement programs. GAO raised a related issue: 
whether or not quality measures comprehensively address the 
measurement needs of the Patient Provider and Affordable Care Act 
(ACA). CMS reports that it has documented how quality measures will be 
used to address all relevant sections of ACA. 

[End of section] 

Appendix VI: Comments from the National Quality Forum: 

National Quality Forum: 
1030 15th Street, NW, Suite 800: 
Washington, Dc 20005
202-783-1300: Main: 
202-783-3434: Fax: 
[Hyperlink, Http://www.qualityforum.org] 

December 9, 2011: 

Linda T. Kohn, Ph.D. 
Director, Health Care: 
U.S. Government Accountability Office: 
441 G Street, NW: 
Washington, DC 20548: 

Dear Dr. Kohn: 

The National Quality Forum (NQF) appreciates the opportunity to 
formally comment on the report, Health Care Quality Measurement: HHS 
Should Address Contractor Performance and Plan for Needed Measures 
(GA0-12-136). Improving the quality and value of our nation's health 
care is vitally important. NQF is committed to encouraging quality 
through performance measurement and accountability. 

We concur with many of the findings in GAO's report. However, we wish 
to clarify and provide context on the findings and recommendations 
regarding the pace of NQF's endorsement activity and cost and timeline 
adjustments in the health information technology (HIT) work. The
novelty and complexity of the work and the consensus-based environment 
in which NQF operates — and which the legislation requires for much of 
the work — all affect timing and cost. 

The report indicates that the performance of particular tasks exceeded 
their initial schedule and cost estimates. As you know, NQF is 
providing its services to HHS under a cost reimbursement contract. 
Authority to use such a contract vehicle is closely regulated. The
Federal Acquisition Regulation ("FAR"), at FAR 16.301-2, specifies 
that such a contract vehicle is to be used only when: 

1. Circumstances do not allow the agency to define its requirements 
sufficiently to allow for a fixed price contract; or: 

2. Uncertainties involved in contract performance do not permit costs 
to be estimated with sufficient accuracy to use any type of fixed 
price contract. 

It is, of course, self-evident that if neither the scope of work nor 
uncertainties regarding contract performance can be reasonably 
estimated, then fixed, immovable schedules should not be expected. The 
deadlines in these schedules were set based upon an early 
understanding of the work. Both HHS and NQF understood that there 
would likely be changes to the work and the deadlines. These factors, 
while identified generally in your report, are so critical to 
understanding the historical performance of the HHS contract that we 
wish to provide specific examples of the diligent efforts expended by 
HHS, NQF, and others to minimize the impact of these factors. 

Specific factors that led to uncertainty regarding the timeline and 
cost of performance under the HHS contract included: 

1. No one previously had attempted to reach cross-industry consensus 
on, endorse, and implement harmonized healthcare quality measures to 
the degree sought under the HHS contract. There was no prior model to 
look to for reliably estimating time and cost. 

2. Reaching consensus required coordination among diverse 
stakeholders, each of which has unique, reasonable needs and 
expectations that were not always consistent with the needs and 
expectations of other stakeholders. All were represented by 
experienced, knowledgeable, often highly opinionated individuals who 
represented their positions effectively. These differences had to be 
resolved through a consensus process. Sometimes this was comparatively 
easy. Many times not. 

3. Ground rules for the contract work sometimes changed mid-effort 
based on the latest legislative or administrative policy requirements 
or recommendations. 

4. No pre-existing commercial software program could be adapted to 
implement the health information technology requirements of the 
contract. New software had to be designed, developed, and tested. 

The potential existence of all of these issues was identified at the 
time of contract award but the extent of their impact could not be 
reasonably estimated. In the paragraphs that follow, we provide some 
specific examples of the issues faced and how HHS, NQF, and others 
addressed the issues and reduced their impact to a minimal level. 

With respect to endorsement activities, a scientifically rigorous, 
consensus-based process is a major part of NQF's value to HHS. Both 
the technical and collaborative nature of that process can adversely 
affect timelines. However, those same characteristics are necessary to 
yield a high quality result that is useful to HHS. 

NQF evaluates measures presented by third parties for NQF endorsement 
against technical criteria for: 

* importance to measure and report; 

* scientific acceptability; 

* feasibility to implement; and; 

* usability of measure results. 

This endorsement process includes review by technical experts, public 
comment, review and voting by multiple NQF committees, and ultimately 
a decision by NQF's Board of Directors to endorse a measure. The 
process is intended to identify strong measures that advance health 
care quality, but deadlines may require adjustment due to: 

* the volume of measures submitted (e.g., NQF received 75 measures for 
consideration in the Child Health Quality Project, approximately twice 
the number of measures typically received in a call for measures); 

* the need for additional information from measure developers (e.g., 
developers were asked to address differences in two competing diabetes 
composite measures in the Patient Outcomes Project, a process that 
took several months); 

* HHS-requested project delays for coordination of federally funded 
measure development (e.g., additional time for federal contractors to 
submit measures in the Nursing Home Project); and; 

* the need to harmonize similar measures. 

Harmonization of measures is a critical and time-consuming part of the 
endorsement process and accounted for modifications to timelines in 
three of the five endorsement projects reviewed by GAO. Harmonizing 
measures involves standardizing specifications or definitions
for measures that have the same focus. For example, in the Patient 
Safety Measures Project, the Centers for Disease Control and 
Prevention and the American College of Surgeons submitted surgical 
site infection measures that differed in terms of inclusions of 
specific procedures and risk adjustment approaches. After consultation 
with NHS, NQF provided the measure developers with the opportunity to 
harmonize their measures into a single surgical site infection 
measure. Other measures in the project moved forward on schedule but 
the project end date was moved out to provide the necessary time for 
NQF to evaluate the newly harmonized measure. The use of related 
measures with different specifications by various public and private 
sector payment and public reporting programs is a source of immense 
frustration and burden for providers and can also be confusing to 
clinicians, consumers and other end users of the performance results. 
Measure harmonization is one of the most important and value-added 
benefits of the NQF endorsement process to HHS and the provider 
community. 

NQF's endorsement work proceeded in parallel with the work of the 
National Priorities Partnership (NPP) and the Measure Applications 
Partnership (MAP), both convened by NQF. The work of NPP and MAP, as 
well as HHS internal planning initiatives, informed NQF's endorsement 
activities and changes to the work and the schedule for endorsement 
resulted, such as the addition of a hospital readmissions project to 
the 2011 work plan to meet legislative mandates. HHS and NQF worked 
together to adjust the timelines where priorities shifted and the 
scope of work was refined. These adjustments are a byproduct of a 
collaborative process — both the collaboration between HHS and NQF and 
the collaboration among stakeholders. 

NQF's work in the health information technology (HIT) arena on measure 
retooling and the Measure Authoring Tool led to similar challenges. 
This work was not just new to NQF. Measure retooling and the Measure 
Authoring Tool were brand new, untested concepts for everyone. NQF, 
HHS, and measure developers learned as they moved through each step of 
the work, and each step informed and modified subsequent steps. To our 
knowledge, no one has comprehensively retooled existing measures prior 
to this effort. 

NQF provided critical technical assistance and coordination to measure 
developers whose measures were scheduled for retooling. NQF does not 
develop performance measures or directly retool measures for use with 
electronic health records. The retooling work under the HHS contract 
was performed by 18 measure developers who own the intellectual 
property in the measures. NQF was deeply enmeshed in supporting the 
efforts of these developers by reviewing the electronic measure 
specifications, identifying errors and inconsistencies, and working 
directly with developers to correctly retool the measures. 

The number of measures that required retooling rose from 10 to 113 in 
six months, a dramatic expansion of NQF's scope of work. In addition, 
the measures HHS identified for retooling shifted several times 
throughout the process. This unavoidably led to higher costs than 
originally planned and revised timelines for completing the work. 

The Measure Authoring Tool is a new software concept designed to 
produce an electronic version of a performance measure for any 
developer who uses it. Given the complexity and the novelty of this 
project, the intent was to create versions of the Tool that would 
evolve by building on prior versions of the Tool. As with any piece of 
software, the Tool is not static and requires constant refinement and 
updates to be able to support HHS' rapidly evolving measurement needs. 
HHS and NQF could not have anticipated the difficulty of this project 
and all of the resources required to move forward, which resulted in 
increased cost and adjustments to timelines. 

Finally, we note that the GAO report states that HHS lacks a 
comprehensive plan for using the work NQF has produced under the 
contract. We believe that HHS is rightly adjusting its plan based upon 
a constant review of the work product, ever-changing national needs, 
and knowledge resulting from an effective collaboration between HHS 
and NQF. We are unaware of a more effective way to approach such a 
massive and important undertaking. The work is innovative and, 
therefore, unpredictable, but HHS and NQF have made, and will continue 
to make, steady progress toward improving America's health care. 

We appreciate the opportunity to review and comment on this report. If 
you have any questions, please do not hesitate to contact me at (202) 
783-1300. 

Sincerely, 

Signed by: 

Janet M. Corrigan, Ph. D.
President and Chief Executive Officer: 

[End of section] 

Appendix VII: GAO Contact and Staff Acknowledgments: 

GAO Contact: 

Linda T. Kohn, (202) 512-7114 or kohnl@gao.gov: 

Staff Acknowledgments: 

In addition to the contact named above, Will Simerl, Assistant 
Director; La Sherri Bush; Krister Friday; Amy Leone; Carla Lewis; John 
Lopez; Elizabeth Martinez; Lisa Motley; Teresa Tucker; Carla Willis; 
and William T. Woods made key contributions to this report. 

[End of section] 

Footnotes: 

[1] Office of the Actuary, Centers for Medicare & Medicaid Services 
(CMS), National Health Expenditures Tables, table 1, downloaded 
09/2/11 from [hyperlink, 
https://www.cms.gov/NationalHealthExpendData/downloads/tables.pdf]. In 
2009, the national spending on health care accounted for 17.6 percent 
of gross domestic product, which totaled $14.1 trillion. The federal 
government share of health care spending for that same year was $678 
billion, or 27 percent. 

[2] GAO, Value in Health Care: Key Information for Policymakers to 
Assess Efforts to Improve Quality While Reducing Costs, [hyperlink, 
http://www.gao.gov/products/GAO-11-445] (Washington, D.C.: July 26, 
2011). 

[3] The Physician Quality Reporting System, for example, is an HHS 
quality reporting program that provides an incentive payment to 
eligible professionals, such as physicians, who satisfactorily report 
data on various quality measures for services that are covered by 
Medicare. 

[4] Value-based purchasing programs are pay-for-performance programs 
that require providers to collect and report information on health 
care quality measures and adjust payment levels based on providers' 
performance against the measures. 

[5] For payment purposes, CMS divides hospital providers into various 
categories, including acute care inpatient hospitals, long-term care 
hospitals, and psychiatric hospitals, among others. The Hospital Value-
Based Purchasing Program applies to acute care inpatient hospitals. 

[6] See 76 Fed. Reg. 2454 (Jan. 13, 2011). PPACA specified that the 
total amount of value-based payments must be equal to the total amount 
of reductions to payments for all hospitals, so the program has a net 
budget-neutral impact. Also see Pub. L. No. 111-148, § 3001, 124 Stat. 
119, 353-62 (2010). 

[7] Pub. L. No. 110-275, § 183, 122 Stat. 2494, 2583-86. The contract 
may be renewed at the end of the 4-year period after a subsequent 
bidding process. 

[8] Pub. L. No. 111-148, § 3014(c), 124 Stat. 119, 387 (2010). 

[9] MIPPA states that GAO's reports shall be submitted no later than 
18 months and 36 months, respectively, after the effective date of the 
contract. 

[10] GAO, Health Care Quality Measurement: The National Quality Forum 
Has Begun a 4-Year Contract with HHS, [hyperlink, 
http://www.gao.gov/products/GAO-10-737] (Washington, D.C.: July 14, 
2010). In this report, we found that (1) NQF had begun work for each 
of the five duties required by MIPPA related to quality measures, (2) 
NQF reported costs and fixed fees totaling approximately $6.5 million 
for the first contract year, and (3) NQF and HHS rely on reviews of 
NQF invoices in order to help ensure that NQF's reported costs are 
proper. 

[11] Under the contract, NQF is required to work with HHS to develop 
an annual work plan for each contract year that outlines the contract 
activities in response to MIPPA. The work plan also delineates 
scheduled time frames for the work NQF is expected to perform that 
contract year and cost estimates for the activities. NQF officials 
told us HHS obligates funds on a yearly basis following HHS's approval 
of the work plan. 

[12] The technical proposal is submitted to HHS and incorporated into 
the scope of work. 

[13] The contract requires that monthly progress reports include NQF's 
accomplishments for the month under each of the contract activities, 
any problems encountered or anticipated and the impact on project time 
frames, and a financial graph that compares NQF's monthly reported 
costs for each of the contract activities with initial cost estimates, 
among other things. 

[14] Direct costs are costs incurred specifically for the contract, 
such as labor costs and payments to subcontractors and consultants. 

[15] Indirect costs are costs that are not directly attributable to a 
specific project or function. These costs cover additional items such 
as employee benefits, overhead, and administrative costs. 

[16] NQF's contract requires that, in accordance with 48 C.F.R. § 
52.216-8, the fixed fee be paid monthly until fee payments reach 85 
percent of the total amount of the fixed fee authorized, and that 
after they reach 85 percent HHS may withhold a reserve up to 15 
percent or $100,000, whichever is less, to protect the government's 
interest. 

[17] NQF and HHS rely on reviews of NQF invoices in order to help 
ensure that NQF's reported costs are proper. NQF officials told us 
that they review the invoices prior to submitting them to HHS and 
carry out other activities, such as using an electronic system to 
track labor hours, in order to help ensure that the costs they report 
in the invoices are proper. HHS also conducts reviews of NQF invoices, 
which are governed by HHS policies and procedures and by requirements 
applicable to federal contracts generally. For more information, see 
[hyperlink, http://www.gao.gov/products/GAO-10-737]. 

[18] Pub. L. No. 111-5, div. A, title XIII, div. B, title IV, 123 
Stat. 115, 226, 467 (2009). 

[19] HHS officials explained that any measure developer can submit a 
measure to be considered for NQF endorsement. Each measure developer 
has its own purpose or plans for implementing a measure, which may or 
may not involve HHS quality and public reporting programs. Therefore, 
all measures submitted may not be applicable to a particular HHS 
health care quality program or initiative. HHS officials told us that 
they will review all the measures received under the contract to 
determine if they are applicable to their health care quality programs 
or initiatives. 

[20] The 2011 National Strategy for Quality Improvement in Healthcare, 
also referred to as the National Quality Strategy, focused on six 
priorities: (1) making care safer; (2) ensuring person-and family-
centered care; (3) promoting effective communication and coordination 
of care; (4) promoting the most effective prevention and treatment of 
the leading causes of mortality, starting with cardiovascular disease; 
(5) working with communities to promote wide use of best practices to 
enable healthy living; and (6) making quality care more affordable. 
The National Quality Strategy was due on January 1, 2011, as required 
by PPACA, and was issued on March 21, 2011. HHS is expected to provide 
annual updates to the report. 

[21] NQF is recognized as a consensus body under the National 
Technology Transfer and Advancement Act of 1995 (NTTAA) and the Office 
of Management and Budget (OMB) Circular A-119, which provides NQF the 
authority needed to facilitate its public/private partnership. OMB 
Circular A-119 defines "consensus" as a "general agreement, not 
necessarily unanimity" so that attempts may be made to resolve 
objections and members may have an opportunity to amend their votes 
after reviewing comments. In addition, the circular states that the 
voluntary consensus process must adhere to the following five 
elements: openness, balance, due process, consensus, and an appeal 
mechanism. See Pub. L. No. 104-113, 110 Stat. 775 (1996). 

[22] NQF does not develop quality measures itself; rather, NQF 
endorses quality measures developed by other organizations, such as 
the Joint Commission, the National Committee for Quality Assurance, 
and the American Medical Association. 

[23] A review for scientific acceptability includes an assessment of 
data from measure testing conducted by measure developers to help 
ensure that the measure produces consistent and credible results about 
the quality of care when implemented. The remaining three criteria NQF 
established for its CDP process are (1) importance to measure and 
report, (2) usability, and (3) feasibility. 

[24] The office of the Assistant Secretary for Planning and Evaluation 
(ASPE)--a component of HHS--is responsible for facilitating the 
workgroup discussions, which occur at least biweekly. 

[25] The contract requires NQF to develop a work plan for each 
contract year that describes the MIPPA projects under the activities 
that NQF is required to perform. The MIPPA workplan includes expected 
time frames and cost estimates for projects under the contract 
activities. The work NQF is required to perform in response to PPACA 
is outlined in a technical proposal submitted to HHS and incorporated 
into the scope of work. 

[26] See 48 C.F.R. §§ 16.301-2, 16.306 (2010). 

[27] For more information on cost-plus-fixed-fee contracts, see GAO, 
Contract Management: Extent of Federal Spending under Cost-
Reimbursement Contracts Unclear and Key Controls Not Always Used, 
[hyperlink, http://www.gao.gov/products/GAO-09-921] (Washington, D.C.: 
Sept. 30, 2009). 

[28] 48 C.F.R. ch.1. The FAR establishes uniform policies for 
acquisition of supplies and services by executive agencies. Agency 
acquisition regulations may implement or supplement the FAR. 

[29] Surveillance includes oversight of a contractor's work to provide 
assurance that the contractor is providing timely and quality goods or 
services and to help mitigate any contract performance problems. 

[30] 48 C.F.R. §§ 44.204(a)(1), 52.244-2 (2010). 

[31] Within CMS, the Office of Acquisition and Grants Management is 
responsible for administering the NQF contract. 

[32] The project officer serves as the technical representative of the 
contracting officer, and provides technical direction to NQF for all 
tasks described in the NQF contract. In addition, the project officer 
monitors NQF's performance and reviews invoices for payment. Another 
HHS official within ASPE, the project manager, also provides support 
to the project officer for the NQF contract. 

[33] The contracting officer enters into, administers, and terminates 
government contracts. The contracting officer negotiates and prepares 
contract documents, modifies terms or conditions of the contract, and 
approves payment of invoices, among other tasks. 

[34] The contract activities and related projects that NQF is expected 
to perform during each year are determined by HHS and NQF, as they 
develop NQF's scope of work. As a result, the number of projects 
varies by year, and some project time frames may extend beyond 1 
contract year. For our reporting period, we determined that NQF is 
conducting work on 63 projects. As of August 31, 2011, of the 63 
projects we identified, 26 have been completed, 34 are in progress, 2 
have been canceled, and 1 has not yet started. See appendix III for 
more details on each of these projects. 

[35] HHS told us that they consider an NQF measure to be endorsed when 
the NQF Board ratifies the measures. However, NQF is required to 
submit a final report to HHS following ratification to complete an 
endorsement project. For purposes of our report, we considered an 
endorsement project to be completed when the board ratifies the 
measures that are the subject of the project. 

[36] The 20 conditions are acute myocardial infarction, Alzheimer's 
disease and related disorders, atrial fibrillation, breast cancer, 
cataract, chronic kidney disease, chronic obstructive pulmonary 
disorder, colorectal cancer, congestive heart failure, diabetes, 
endometrial cancer, glaucoma, hip/pelvic fracture, ischemic heart 
disease, lung cancer, major depression, osteoporosis, prostate cancer, 
rheumatoid arthritis and osteoarthritis, and stroke/transient ischemic 
attack. 

[37] NQF completed the child health quality project by endorsing three 
additional child health quality measures in September 2011, as 
scheduled. As of November 2011, NQF was continuing to conduct work on 
the patient safety project, such as through the endorsement of an 
additional patient safety measure in September 2011. NQF expected to 
complete additional review of measures under this project by December 
2011. 

[38] Mathematica reviewed 23 projects in its analysis, including 7 
endorsement projects conducted under the HHS contract. 

[39] Under NQF's maintenance process, NQF can retire a measure, or 
remove its endorsement status for various reasons, such as if the 
board determines a measure is obsolete. 

[40] Although these measures were reviewed under the contract, they 
were not reviewed under NQF's updated Endorsement Maintenance Process 
because the reviews began prior to its implementation. 

[41] During our reporting period, NQF maintained 80 measures using its 
time-limited endorsement review process in areas such as infectious 
diseases and neurology. NQF also maintained 3 measures under its ad 
hoc review process related to various conditions, including patient 
safety. 

[42] As of November 2011, NQF was still completing the cardiovascular 
and surgery projects and expected to complete the work by December 
2011 and January 2012, respectively, as scheduled. 

[43] HHS officials told us that they awarded contracts to Mathematica 
Policy Research, Inc., and Abt Associates to conduct feasibility 
testing on 69 of 113 retooled measures. These contracts were awarded 
on June 8, 2011, and July 8, 2011, respectively. For the remaining 44 
retooled measures, HHS officials told us that HHS issued a 
solicitation for a contract to conduct feasibility testing of the 
measures; however, they did not receive any bids. As of December 2011, 
they said that they will not be issuing a solicitation for a new 
contract to test the 44 retooled measures. 

[44] As of October 2011, NQF completed testing of the beta version 
that was made available for public use in September 2011 and expected 
to publish an updated version in January 2012, as scheduled. 

[45] The National Priorities Partnership (NPP) represents over 48 
major national organizations from the public and private sectors, 
including consumers, purchasers, clinicians, and communities. Under 
PPACA, HHS is required to solicit annual input from such a group on 
the National Quality Strategy. 

[46] On October 4, 2011, NQF reported that it submitted these reports 
to HHS on October 1, 2011, as scheduled. 

[47] Regionalized emergency medical care refers to directing patients 
to emergency facilities with optimal capabilities for a given type of 
illness or injury to help coordinate emergency care across a region. 

[48] As of August 2011, three of these five projects had been 
completed. However, NQF was still completing the patient safety 
project, which was delayed approximately 10 months to allow for 
harmonization of measures, and the child health quality project. 

[49] Harmonization is a process where related measures are 
standardized in a manner by which they can be used in multiple 
settings when relevant. NQF defines "related measures" as those that 
either (1) address the same concepts for measure focus (target 
process, condition, event, outcome) but a different target population; 
or (2) address different concepts for measure focus (target process, 
condition, event, outcome) but the same target population. NQF 
officials stated in some instances NQF needed to facilitate 
harmonization discussions between measure developers to help ensure 
that the most appropriate measure was being used, and that the measure 
portfolio did not include duplicative measures, and to avoid confusion 
within the health care community. 

[50] Two of the five projects that have not met expected time frames 
are still in progress and are expected to be completed by the end of 
the third contract year. The remaining three have been completed. 

[51] Resource-use measures compare resources used (i.e., cost of care) 
with either (1) a specific population, (2) a specific service, or (3) 
a series of related services over time, to help identify the resources 
necessary for delivering high-quality care and to promote greater 
efficiency in delivery of health care services. 

[52] Geographic-level efficiency measures would be used for quality 
measure reporting at the geographic and population level. 

[53] HHS officials told us that they accepted the final version of 
this paper in October 2011. 

[54] NQF officials told us that it was originally required to retool 
44 measures under the contract; however, HHS expanded the scope of 
work in 2010 to 113 measures. This required additional labor and other 
resources that increased the costs associated with the retooling 
project. 

[55] Our previous work shows that without the ability to generate 
reliable cost estimates, programs are at risk of experiencing cost 
overruns, missed deadlines, and performance shortfalls. GAO, GAO Cost 
Estimating and Assessment Guide: Best Practices for Developing and 
Managing Capital Program Costs, [hyperlink, 
http://www.gao.gov/products/GAO-09-3SP] (Washington, D.C.: March 2009). 

[56] The requirement in the contract to conduct annual performance 
evaluations is consistent with a FAR provision that provides that 
agencies may specify that interim performance evaluations be performed 
for multiyear contracts. See 48 C.F.R. § 42.1502(a) (2010). 

[57] According to HHS, total funding obligated was approximately $13.6 
million for the second contract year, which covered January 14, 2010, 
through January 13, 2011. 

[58] According to HHS, total funding obligated was approximately $25 
million for the third contract year, which covers January 14, 2011, 
through January 13, 2012. 

[59] As of August 2011, NQF had reported costs and fixed fees totaling 
$28.9 million since the contract was awarded in January 2009. 
Specifically in our July 2010 report, we highlighted that NQF reported 
costs and fixed fees totaling approximately $6.5 million for the first 
contract year--January 14, 2009, to January 13, 2010. In addition, NQF 
had reported about $22.4 million in costs and fixed fees from January 
14, 2010, through August 31, 2011--our reporting period that covers 
the second and part of the third contract years. See [hyperlink, 
http://www.gao.gov/products/GAO-10-737]. 

[60] From January 14, 2011, through August 31, 2011, NQF reported 
about $4 million in costs and fixed fees for PPACA-required 
activities. HHS officials told us that they plan to obligate $10 
million of the PPACA funding for health care quality measurement each 
fiscal year from 2011 through 2014. Therefore, at least $40 million in 
PPACA funding is expected to be available to NQF under the contract. 
HHS officials said that because PPACA does not specify the proportion 
of the $100 million for NQF's and HHS's health care quality 
measurement work for NQF, they determined that $10 million per year 
was a reasonable amount. 

[61] For the purpose of this report, we defined a measure as used if 
the measure was identified by HHS and included in an HHS program or 
initiative as of August 31, 2011, and indicated as such by a final 
rule or other HHS documentation. We defined a measure as planned for 
use if the measure was identified by HHS for potential use in an HHS 
program or initiative as of August 31, 2011, and indicated as such by 
a proposed rule or other HHS documentation. 

[62] Measures that are endorsed, maintained, and retooled under the 
NQF contract are counted separately, even if they are the same 
measure. Therefore, the 344 measures HHS has received from NQF do not 
represent 344 distinct measures. For example, a measure that had been 
maintained under the contract and later retooled would be counted as 
two separate measures. Therefore, the measure would appear twice in 
our count of the total number of measures received from NQF--once as a 
maintained measure and then separately as a measure that was retooled. 
Similarly, the measure would also be counted as two separate measures 
for purposes of determining the number used or planned for use. If 
both versions of the measure were used or planned for use in HHS 
programs or initiatives, then it would also appear twice in our count 
of measures used or planned for use. 

[63] See 75 Fed. Reg. 44,314 (July 28, 2010). The Medicare and 
Medicaid EHR Incentive programs will provide incentive payments to 
eligible Medicare and Medicaid professionals and hospitals that adopt 
and use health information technology, and reduce payments for those 
Medicare professionals or hospitals that do not. The Medicare and 
Medicaid EHR programs are divided into three stages. Stage I, which is 
underway, focuses on establishing the functionalities in certified EHR 
technology that will allow for continuous quality improvement and ease 
of information exchange. Stages II and III will potentially become 
more stringent as the capabilities of health information technology 
infrastructure increase. In 2011, part of the requirement for Stage I 
of the Medicare EHR Program is that eligible professionals attest that 
they have met the program's reporting requirements, which includes 
that they report on a certain number of measures that were retooled by 
NQF. 

[64] HHS officials said that the 44 measures are not included as part 
of the scope of work under the two contracts under which feasibility 
testing of the retooled measures will be performed. 

[65] See, e.g., Pub. L. No. 111-148, § 3004(a), 124 Stat. 119, 368 
(2010) (codified as amended at 42 U.S.C. § 1395ww(m)(5)) (directing 
HHS to publish measures for long-term care hospitals and requiring the 
department to consider measures endorsed by NQF). For specific areas 
for which NQF has not endorsed feasible and practical measures, HHS 
may specify measures that have not been endorsed by NQF as long as due 
consideration is given to measures that have been endorsed or adopted 
by a consensus organization. 

[66] The NPP identified six priorities in a November 2008 report: (1) 
patient and family engagement, (2) population health, (3) patient 
safety, (4) care coordination, (5) palliative and end-of-life care, 
and (6) elimination of overuse. In its October 2010 report to HHS, the 
NPP identified two additional priority areas: equitable access to 
ensure that all patients have access to affordable, timely, and high-
quality care; and infrastructure supports, such as health information 
technology (IT), to address underlying system changes necessary to 
attain the goals of the other priority areas. See National Priorities 
Partnership, Input To The Secretary Of Health And Human Services On 
Priorities For The 2011 National Quality Strategy (Washington, D.C.: 
2010). 

[67] CMS is an agency within HHS. 

[68] The Measures Management System is used by divisions within CMS 
and provides guidance to CMS-contracted measure developers on the 
development and maintenance of measures. It was developed to help CMS 
manage an ever-increasing demand for quality measures to use in its 
various public reporting and quality programs as well as in value-
based purchasing initiatives. 

[69] The technical expert panel consists of a group of recognized 
experts in relevant fields including clinicians, statisticians, 
quality improvement experts, methodologists, and pertinent measure 
developers who are recruited to provide input on the measure under 
development by the contract. 

[End of section] 

GAO’s Mission: 

The Government Accountability Office, the audit, evaluation, and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the 
performance and accountability of the federal government for the 
American people. GAO examines the use of public funds; evaluates 
federal programs and policies; and provides analyses, recommendations, 
and other assistance to help Congress make informed oversight, policy, 
and funding decisions. GAO’s commitment to good government is 
reflected in its core values of accountability, integrity, and 
reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO’s website [hyperlink, http://www.gao.gov]. Each 
weekday afternoon, GAO posts on its website newly released reports, 
testimony, and correspondence. To have GAO e mail you a list of newly 
posted products, go to [hyperlink, http://www.gao.gov] and select “E-
mail Updates.” 

Order by Phone: 

The price of each GAO publication reflects GAO’s actual cost of 
production and distribution and depends on the number of pages in the 
publication and whether the publication is printed in color or black 
and white. Pricing and ordering information is posted on GAO’s 
website, [hyperlink, http://www.gao.gov/ordering.htm]. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or 
TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card, 
MasterCard, Visa, check, or money order. Call for additional 
information. 

Connect with GAO: 

Connect with GAO on facebook, flickr, twitter, and YouTube.
Subscribe to our RSS Feeds or E mail Updates. Listen to our Podcasts.
Visit GAO on the web at [hyperlink, http://www.gao.gov]. 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 
Website: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]; 
E-mail: fraudnet@gao.gov; 
Automated answering system: (800) 424-5454 or (202) 512-7470. 

Congressional Relations: 

Ralph Dawn, Managing Director, dawnr@gao.gov, (202) 512-4400
U.S. Government Accountability Office, 441 G Street NW, Room 7125
Washington, DC 20548. 

Public Affairs: 
Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800
U.S. Government Accountability Office, 441 G Street NW, Room 7149 
Washington, DC 20548.