Research Reliability: Federal Actions Needed to Promote Stronger Research Practices
While scientific discoveries can be serendipitous, reliable research is more often built on successive studies that employ rigorous design, documentation, and confirmation.
The U.S. government funded over $42 billion in science research in 2019. How can federal agencies help assure the reliability of that research?
Experts and others have suggested that agencies implement practices such as disclosing results and methods earlier, as well as incentivizing researchers to use rigorous methods and make their research more transparent.
We recommended that agencies collect information to learn how best to implement such practices.
What GAO Found
The National Institutes of Health (NIH), the National Science Foundation (NSF), and the National Aeronautics and Space Administration (NASA) are the three largest federal funders of basic scientific research in the United States. According to leading experts GAO interviewed, these agencies could do more to increase the rigor and transparency of the research they fund by taking actions to better align awards and recognition for researchers with more rigorous and transparent research practices. Experts suggested, for example, that agencies could incentivize or mandate that researchers preregister their studies as a means to share their research plans before the research is conducted. Doing so would enable other researchers to comment on and strengthen the methodology and analysis plans. Experts further suggested that agencies help improve standards for data repositories where research data are stored publicly, encourage the publication of null research results, and support training in statistical analysis and study design. Although the scientific community has developed many such practices to enhance research reliability, GAO found that they are not widely adopted because of researcher misconceptions and misaligned incentives in funding and publishing, among other things.
Role of Rigor and Transparency in Research Reliability
NIH, NSF, and NASA have taken steps to promote and support additional rigor and transparency in research, such as establishing requirements for researchers to disclose research results and associated data publicly. However, these agencies largely rely on grant application reviews and the prepublication peer review process to help ensure research rigor. GAO found that these agencies do not evaluate the rigor and transparency of the research they fund to help identify strategies for improvement. Specifically, they do not collect indicators of rigorous study design and transparency of research results such as study sample size, adherence to research plans, or the extent to which research data are findable, accessible, and usable. As a result, the agencies lack information to support changes to the grant making process and research funding priorities. Federal guidance and Standards for Internal Control in the Federal Government call for agencies to prioritize making federally funded research more rigorous and transparent and to use quality information to achieve agency objectives. Without this information on the research they fund, agencies are limited in their ability to take effective actions to improve research reliability, like those the experts described to GAO.
Why GAO Did This Study
In 2019, the U.S. government funded more than $42 billion in basic scientific research across a wide range of scientific disciplines. Unsuccessful attempts to reproduce and replicate research results have been documented across many scientific disciplines, including those funded by NASA, NIH, and NSF. The scientific community has expressed concern over the difficulty of replicating prior research results.
GAO was asked to review strategies to improve the reliability of federally funded research. Among other things, this report (1) examines what actions, according to experts, federal agencies could take to foster rigor and transparency in the research they fund; and (2) assesses the extent to which selected federal science funding agencies have taken actions to improve rigor and transparency. GAO conducted a literature review; reviewed NIH, NSF, and NASA documents; and conducted four roundtable discussions with 22 experts. GAO also interviewed agency officials as well as stakeholders from academia, professional societies, publishing, and other parts of the scientific community.
Recommendations
GAO is making six recommendations, two each to NIH, NSF, and NASA to evaluate research using indicators of rigor and transparency, and to use this information to inform further actions. NIH and NSF concurred with the recommendations. NASA did not concur with our first recommendation and partially concurred with our second. GAO continues to believe the recommendations are valid.
Recommendations for Executive Action
| Agency Affected | Recommendation | Status |
|---|---|---|
| National Institutes of Health | The Director of NIH should collect information on relevant indicators of rigor to assess the research projects the agency funds, and implement steps, as needed, to promote strong research practices in future work. (Recommendation 1) |
When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.
|
| National Institutes of Health | The Director of NIH should take steps to collect information to determine whether current policies and requirements are adequate to achieve transparency by ensuring research results and data are findable, accessible, and usable, and implement programmatic or policy changes, if needed. (Recommendation 2) |
When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.
|
| National Science Foundation | The Director of NSF should collect information on relevant indicators of rigor to assess the research projects the agency funds, and implement steps, as needed, to promote strong research practices in future work. (Recommendation 3) |
When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.
|
| National Science Foundation | The Director of NSF should take steps to collect information to determine whether current policies and requirements are adequate to achieve transparency by ensuring research results and data are findable, accessible, and usable, and implement programmatic or policy changes, if needed. (Recommendation 4) |
When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.
|
| National Aeronautics and Space Administration | The Administrator of NASA should collect information on relevant indicators of rigor to assess the research projects the agency funds, and implement steps, as needed, to promote strong research practices in future work. (Recommendation 5) |
When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.
|
| National Aeronautics and Space Administration | The Administrator of NASA should take steps to collect information to determine whether current policies and requirements are adequate to achieve transparency by ensuring research results and data are findable, accessible, and usable, and implement programmatic or policy changes, if needed. (Recommendation 6) |
When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.
|