Improvements Needed for Measuring Awardee Performance in Meeting Medical and Public Health Preparedness Goals
GAO-13-278: Published: Mar 22, 2013. Publicly Released: Mar 22, 2013.
What GAO Found
Available measures and awardee data provide some evidence that Hospital Preparedness Program (HPP) awardees have generally made progress in carrying out activities to achieve medical preparedness goals; however, the Department of Health and Human Services (HHS) Office of the Assistant Secretary for Preparedness and Response (ASPR) lacked a comprehensive performance management system to fully assess awardee progress. According to prior GAO work and the GPRA Modernization Act of 2010 (GPRAMA), successful performance measurement systems should include a select set of performance measures tied to realistically achievable targets with clearly defined milestones. GAO's analysis of ASPR data showed general progress. For example, the percentage of all 62 awardees' participating hospitals with medical evacuation and shelter-in-place plans increased from 79.9 percent to 88.3 percent from fiscal year 2007 to fiscal year 2011. However, while ASPR collected data on a range of activities, it did not have consistent performance measures and targets in place across this entire period. Beginning with fiscal year 2012, ASPR developed new provisional performance measures for the eight new capabilities that awardees are to use for HPP planning for the next 5 years and set 5-year targets for these measures. However, it did not develop annual milestones, which may make it difficult for ASPR and awardees to assess incremental progress toward meeting HPP goals.
Although Public Health Emergency Preparedness (PHEP) program awardees are improving in their ability to carry out preparedness activities, HHS's Centers for Disease Control and Prevention (CDC) lacked a consistent set of performance measures and targets to adequately assess the degree of awardee progress toward meeting the PHEP goal. For example, from fiscal years 2007 through 2011, the number of measures ranged from 5 to 30 in any one year, and CDC had only four targets for any of them. GAO's analysis of CDC data showed general progress on the measures. For example, for one measure, the average time it took the 62 awardees to assemble appropriate response staff decreased from 35 minutes in 2007 to 31 minutes in 2011, although the 50 state awardees did not always meet the 60-minute target that CDC set for them starting in 2009. Beginning with fiscal year 2012, CDC released 47 provisional performance measures for 14 of the 15 new PHEP capabilities but developed only four associated targets. Without consistent performance measures and associated targets, in accordance with prior GAO work and GPRAMA, CDC may not be able to assess how awardees are making progress toward meeting the PHEP goal.
HHS uses internal databases, site visits, and audit reports to help awardees meet HPP and PHEP application, financial, and reporting requirements. ASPR and CDC use internal databases to generate reports on awardee progress in meeting application renewal and reporting deadlines, to assess application completeness, and to periodically query databases in order to review financial information. They also conduct regular site visits and review state and federal audit reports to help awardees meet program requirements and assess awardees' use of funds. ASPR and CDC require awardees that have problems managing their HPP or PHEP funds to complete corrective action plans, and they restrict awardees' access to funds in more serious cases.
Why GAO Did This Study
HHS provides funding to the 50 states and 12 municipalities, territories, and freely associated states, primarily through ASPR's HPP and CDC's PHEP cooperative agreements, to help them build their capability to respond to emergencies such as hurricanes, pandemics, or terrorist events. The 62 awardees are to use this funding to help achieve the HPP goals of strengthening hospital preparedness and medical surge capacity and the PHEP goal of strengthening public health preparedness, and they must meet certain application, financial, and reporting requirements. GAO was asked about the effects of federal support on state and local response capabilities. GAO (1) assessed awardee progress in meeting HPP goals and how ASPR measures that progress, (2) assessed awardee progress in meeting the PHEP goal and how CDC measures that progress, and (3) identified the mechanisms HHS uses to ensure that awardees are meeting application, financial, and reporting requirements. GAO reviewed HPP and PHEP guidance, performance measures, and other documents; interviewed HHS officials; and analyzed HPP and PHEP data for fiscal years 2007 through 2011.
What GAO Recommends
GAO recommends that (1) HHS develop objective and quantifiable performance targets and incremental milestones tied to HPP and PHEP performance measures and (2) ensure that measures remain consistent and comparable to sufficiently measure progress. HHS generally agreed with GAO's recommendations but indicated that it would not be able to fully implement them for several years.
For more information, contact Marcia Crosse at (202) 512-7114 or firstname.lastname@example.org.
Recommendations for Executive Action
Comments: Since we examined the HPP and PHEP cooperative agreements in 2012, ASPR and CDC have reported plans to develop performance targets and incremental milestones that correspond to HPP and PHEP performance measures. In a July 2014 update, HHS officials noted that for the HPP program, agency officials expected to develop new targets and milestones in fiscal years 2015-2016. For the PHEP program, officials stated that CDC planned to introduce evidence-based targets for several key performance measures through 2014 and possibly later. The agency also noted that, as retained measures continued to stabilize, CDC could set interim targets for some measures, along with targets based on individual awardee benchmarks. In August 2015, HHS told us that in collaboration with CDC, ASPR has collected performance measure data and planned to keep its performance measures constant through the remainder of the project period (2017). The agency reported that, based on collected performance data, it will establish benchmarks to measure the incremental progress of the awardees; the most recent HPP measures and guidance document provided by HHS (for the 2013- 2014 period) did not include performance targets or incremental milestones for the measures or indicators. HHS did not provide an update regarding PHEP performance measure development. As of September 2015, we are awaiting documentation of the PHEP targets and any incremental milestones. We will update the status of this recommendation when we receive additional information.
Recommendation: To help ensure that HHS is adequately and comprehensively assessing HPP and PHEP awardees' performance and progress in meeting the medical and public health preparedness goals of the cooperative agreements, the Secretary of Health and Human Services should direct ASPR and CDC to develop objective and quantifiable performance targets and incremental milestones that correspond to the new HPP and PHEP performance measures, against which HHS can gauge progress toward the medical and public health preparedness goals of the cooperative agreements and direct technical assistance, as needed.
Agency Affected: Department of Health and Human Services
Comments: Since we examined the HPPS and PHEP cooperative agreements in 2012, ASPR and CDC have continued to make changes to the performance evaluation programs. For example, while ASPR planned to maintain its existing two HPP program measures to help ensure consistency across the current project period, it planned to use the remaining budget years in the project period (through 2017) to further hone, revise, and finalize HPP indicators for each HPP capability, which could result in additional changes, and to collect baseline data to develop incremental targets and milestones. In a July 2014 update, HHS stated that PHEP FY 2014 changes included a reduction from 28 to 15 performance measures and from 3 to a single evaluation tool. Two of the 15 performance measures would be in place for 2014 only, and for FY 2015 the program would have 13 performance measures and 1 evaluation tool, which were expected to remain in place for the remaining years of the project period. PHEP targets were expected to continue to evolve, and CDC was also developing a broader PHEP evaluation framework to include other methods, outside of performance measures, to monitor and evaluate awardee programs. In August 2015, HHS provided us an HPP measures and guidance document that shows that, as planned, the refined measures remained constant from the previous budget period at two overall measures, called program measures, each with seven indicators; however, no information regarding current targets was included in the document. Additionally, HHS did not provide a corresponding PHEP measures and guidance document. As of September 2015, we are awaiting documentation to show the updated PHEP measures, targets, and any incremental milestones, and any information regarding HPP targets. We will update the status of this recommendation when we receive additional information.
Recommendation: To help ensure that HHS is adequately and comprehensively assessing HPP and PHEP awardees' performance and progress in meeting the medical and public health preparedness goals of the cooperative agreements, the Secretary of Health and Human Services should ensure that performance measures and targets remain consistent across the 5-year project cycle and that any future measures be comparable to determine whether awardees are making progress toward meeting short- and long-term medical and public health preparedness goals of the cooperative agreements.
Agency Affected: Department of Health and Human Services