Infectious Disease Modeling:
Opportunities to Improve Coordination and Ensure Reproducibility
GAO-20-372: Published: May 13, 2020. Publicly Released: Jun 4, 2020.
Additional Materials:
- Highlights Page:
- Full Report:
- Accessible Version:
Contact:
(202) 512-6888
personst@gao.gov
Office of Public Affairs
(202) 512-4800
youngc1@gao.gov
Outbreaks of infectious diseases—such as novel coronavirus and pandemic flu—have raised concerns about how federal agencies use modeling to predict a disease’s course. Models can help decision makers set disease control policies and allocate resources. If models are unsound, they may not produce the reliable predictions needed to make good decisions.
We examined how Health and Human Services, which includes the Centers for Disease Control and Prevention, uses and assesses models. We recommended that HHS improve coordination of modeling across agencies and ensure models are reproducible, which helps build confidence in their results.
An HHS employee working on Infectious Disease Planning and Response
Staff, computers, monitors, and projected screens
Additional Materials:
- Highlights Page:
- Full Report:
- Accessible Version:
Contact:
(202) 512-6888
personst@gao.gov
Office of Public Affairs
(202) 512-4800
youngc1@gao.gov
What GAO Found
Within the Department of Health and Human Services (HHS), the Centers for Disease Control and Prevention (CDC) and the Office of the Assistant Secretary for Preparedness and Response (ASPR) used models to inform decision-making during and after outbreaks of Ebola, Zika, and pandemic influenza. These agencies' modeling efforts informed public health planning, outbreak response, and, to a limited extent, resource allocation. Four CDC centers perform modeling.
HHS agencies reported using multiple mechanisms to coordinate modeling efforts across agencies, but they do not routinely monitor, evaluate, or report on the extent and success of coordination. Consequently, they risk missing opportunities to identify and address modeling challenges—such as communicating clearly, and obtaining adequate data and resources—before and during an outbreak. As a result, agencies may be limiting their ability to identify improvements in those and other areas. Further, there is potential for overlap and duplication of cross-agency modeling efforts, which could lead to inefficiencies.
Office of the Assistant Secretary for Preparedness and Response's Visualization Hub, which Can Be Used for Infectious Disease Planning and Response
CDC and ASPR generally developed and assessed their models in accordance with four steps GAO identified as commonly-recognized modeling practices: (1) communication between modeler and decision maker, (2) model description, (3) verification, and (4) validation. However, for four of the 10 models reviewed, CDC did not provide all details needed to reproduce model results, a key step that lets other scientists confirm those results. GAO found that CDC's guidelines and policy do not address reproducibility of models or their code. This is inconsistent with HHS guidelines and may jeopardize the reliability of CDC's research.
This report also identifies several modeling-related challenges, along with steps agencies have taken to address them.
Why GAO Did This Study
Outbreaks of infectious diseases—such as Ebola, Zika, and pandemic influenza—have raised concerns from Congress about how federal agencies use modeling to, among other things, predict disease distribution and potential impacts. In general, a model is a representation of reality expressed through mathematical or logical relationships. Models of infectious diseases can help decision makers set policies for disease control and may help to allocate resources.
GAO was asked to review federal modeling for selected infectious diseases. This report examines (1) the extent to which HHS used models to inform policy, planning, and resource allocation for public health decisions; (2) the extent to which HHS coordinated modeling efforts; (3) steps HHS generally takes to assess model development and performance; and (4) the extent to which HHS has addressed challenges related to modeling. GAO reviewed documents and interviewed HHS officials, state officials, and subject matter experts. GAO identified practices commonly used to assess infectious disease model performance and reviewed 10 selected modeling efforts to see if they followed these practices.
What GAO Recommends
GAO recommends that HHS (1) develop a way to routinely monitor, evaluate, and report on modeling coordination efforts across multiple agencies and (2) direct CDC to establish guidelines to ensure full reproducibility of its models. HHS agreed with GAO's recommendations.
For more information, contact Timothy M. Persons, PhD, Chief Scientist at (202) 512-6888 or personst@gao.gov.
Recommendations for Executive Action
Status: Open
Comments: The Department of Health and Human Services concurred with this recommendation and stated that it is developing a process whereby it will coordinate its efforts in infectious disease modeling across its components, including efforts to monitor, evaluate and report on its coordination. When we confirm what actions the agency has taken in response to the recommendation, we will provide updated information.
Recommendation: The Secretary of Health and Human Services should develop a mechanism to routinely monitor, evaluate, and report on coordination efforts for infectious disease modeling across multiple agencies. (Recommendation 1)
Agency Affected: Department of Health and Human Services
Status: Open
Comments: The Centers for Disease Control and Prevention concurred with this recommendation. When we confirm what actions the agency has taken in response to the recommendation, we will provide updated information.
Recommendation: The Secretary of Health and Human Services should direct CDC to establish guidelines that ensure full reproducibility of CDC's research by sharing with the public all permissible and appropriate information needed to reproduce research results, including, but not limited to, model code. (Recommendation 2)
Agency Affected: Department of Health and Human Services
Explore the full database of GAO's Open Recommendations
»
Mar 1, 2021
-
Medicare Part B:
Payments and Use for Selected New, High-Cost DrugsGAO-21-252: Published: Mar 1, 2021. Publicly Released: Mar 1, 2021.
Feb 24, 2021
-
Covid-19:
Key Insights from GAO's Oversight of the Federal Public Health ResponseGAO-21-396T: Published: Feb 24, 2021. Publicly Released: Feb 24, 2021.
Feb 17, 2021
Feb 11, 2021
-
Operation Warp Speed:
Accelerated COVID-19 Vaccine Development Status and Efforts to Address Manufacturing ChallengesGAO-21-319: Published: Feb 11, 2021. Publicly Released: Feb 11, 2021.
Feb 5, 2021
-
DOD Health Care:
DOD Should Monitor Implementation of Its Clinical Practice GuidelinesGAO-21-237: Published: Feb 5, 2021. Publicly Released: Feb 5, 2021. -
Medicaid:
CMS Needs to Implement Risk-Based Oversight of Puerto Rico's Procurement ProcessGAO-21-229: Published: Feb 5, 2021. Publicly Released: Feb 5, 2021. -
VA Health Care:
Community Living Centers Were Commonly Cited for Infection Control Deficiencies Prior to the COVID-19 PandemicGAO-21-195R: Published: Jan 6, 2021. Publicly Released: Feb 5, 2021.
Jan 28, 2021
-
COVID-19:
Critical Vaccine Distribution, Supply Chain, Program Integrity, and Other Challenges Require Focused Federal AttentionGAO-21-265: Published: Jan 28, 2021. Publicly Released: Jan 28, 2021.
Jan 21, 2021
-
Rural Hospital Closures:
Affected Residents Had Reduced Access to Health Care ServicesGAO-21-93: Published: Dec 22, 2020. Publicly Released: Jan 21, 2021. -
Health Care Funding:
Federal Obligations to and Funds Received by Certain Organizations Involved in Health-Related Services, 2016 through 2018GAO-21-188R: Published: Dec 21, 2020. Publicly Released: Jan 21, 2021.
Looking for more? Browse all our products here