Federal Research: Additional Actions Needed to Improve Public Access to Research Results

Fast Facts

Public access to the results of federally funded research can accelerate scientific breakthroughs. In 2013, certain federal agencies were directed to create plans for increasing access to publications and data they funded.

The 19 agencies we reviewed made progress, but some have not fully implemented their plans. For example:

7 agencies have not taken steps to make data findable, such as creating a single web access point

4 don’t require all researchers to submit a plan to provide access to data

11 don’t fully ensure that researchers comply with access requirements

We made 37 recommendations to 16 agencies to address these and other issues.

Hands on a keyboard

Highlights

What GAO Found

The 19 agencies that GAO reviewed have made progress implementing their plans to increase public access to federally funded research results (publications and data), as called for in a 2013 Office of Science and Technology Policy (OSTP) memorandum. However, some agencies have not fully implemented some aspects of their plans, in particular those related to data access and mechanisms to ensure researchers comply with public access requirements.

Examples of Agencies' Progress Implementing Plans to Increase Public Access to Federally Funded Research Results

Public access plan topic	Extent of agency progress
Repositories	All 19 agencies have identified federally owned or managed locations, known as repositories, for preservation and public access to publications. For data, agencies rely on an array of federal and nonfederal repositories. However, seven agencies have not taken steps, such as establishing a single web-based point of access, or have not fully implemented plans to help the public find data stored across repositories. Taking such steps could better support public access to federally funded data.
Data management plans (DMPs)	Sixteen of 19 agencies reported requiring researchers to submit a DMP, which is supposed to describe how researchers will provide for long-term preservation and access to data they generate, or a justification for why that cannot be done. However, four agencies reported they have not established such requirements or have done so on a limited basis. Without requiring DMPs from agency-funded researchers, agencies may not be able to ensure that agency-funded data are being made publicly available.
Compliance	Eleven agencies reported that they have not fully developed or implemented mechanisms to ensure researchers comply with applicable public access requirements. Officials cited several reasons for this, including resource constraints and difficulty with tracking and measuring compliance. Without fully implementing compliance mechanisms—as called for in the OSTP memorandum—agencies may not have assurance that all appropriate federally funded research results are being made publicly available.

Source: GAO analysis of agency public access plan implementation efforts. | GAO-20-81

Agencies are coordinating with each other and with nonfederal stakeholders to implement public access plans, including through an interagency group led by OSTP and five other agencies. However, the group has not fully implemented selected leading practices identified by GAO that can enhance and sustain interagency collaboration, such as defining and articulating common outcomes. For example, according to OSTP staff, key outcomes have not yet been decided upon. Agency officials and stakeholders identified several challenges to implementing public access plans that interagency coordination might help them address, such as

Absence of common standards in several areas;

Measuring effectiveness of public access plan implementation; and

Balancing providing public access with safeguarding sensitive information.

By taking steps to fully implement relevant leading collaboration practices, the interagency group could help agencies better marshal their collective efforts to address common challenges to public access plan implementation.

Why GAO Did This Study

Research and development helps catalyze breakthroughs that improve the overall health and wellbeing of our society. Federal research and development expenditures averaged about $135 billion annually for fiscal years 2015 to 2017. According to OSTP, providing free public access to federally funded research results can improve both the impact and accountability of this important federal investment. In February 2013, OSTP directed federal agencies with more than $100 million in annual research and development expenditures to develop a plan to support increased public access to the results of federally funded research. GAO was asked to examine public access to federally funded research results. This report examines the extent of agencies' (1) progress implementing plans to increase public access to federally funded research results and (2) coordination on public access plan implementation. GAO administered a questionnaire to 19 federal agencies selected based on annual research and development expenditure amounts, among other criteria; reviewed agency documents; and interviewed officials from 11 agencies, OSTP, and 21 stakeholder organizations.

Recommendations

GAO is making 37 recommendations to 16 agencies to promote full and effective implementation of agency public access plans. For example, GAO recommends that OSTP and 5 agencies leading a public access interagency group take steps to fully implement selected leading collaboration practices. Of the 16 agencies, 15 agreed with GAO's recommendations while 1 (OSTP) disagreed. GAO continues to believe the recommendation to OSTP is warranted.

Recommendations for Executive Action

Agency Affected	Recommendation	Status
Department of Defense	The Secretary of Defense should take steps to ensure appropriate agency-funded research data are readily findable and accessible to the public. (Recommendation 1)	Open The Department of Defense partially concurred with this recommendation noting the challenge with balancing ensuring public access to research data with considerations of national security and personally identifiable information. As discussed in our report, our recommendation to DOD regarding findability and accessibility of agency-funded research data was qualified to pertain to appropriate agency-funded research data--recognizing that it might not be appropriate to make certain datasets publicly available because of national security or other concerns. In May 2021, DOD provided an update stating it had issued guidance on the responsible release of DOD data, which includes initiating a study of existing pre-publication and security review policies as well as a partnership between the DOD Data Information Officer and others to formulate tools and techniques to better understand and counter the risks of data aggregation. DOD provided the guidance document in May 2023. As of June 2023, DOD provided an update in which it indicated that it is still working to develop methodologies to identify situations where aggregation of multiple sets of unclassified data could reveal national security vulnerabilities. We will provide an update when DOD provides additional information on steps taken to implement this recommendation.
Department of Education	The Secretary of Education should take steps to ensure appropriate agency-funded research data are readily findable and accessible to the public. (Recommendation 2)	Closed – Implemented The Department of Education concurred with this recommendation. According to its response to our report, the Department awarded a contract to support enhancements to its Education Resources Information Center (ERIC) to link scholarly research publications supported by the Department to their publicly accessible datasets. According to information Education provided, all grantees who submit through the ERIC submission system will be asked to submit a link to their data management plan, which includes a link to the underlying data. In addition, Education stated that, as part of an effort to minimize the burden for grantees and contractors, it negotiated agreements with the publishers of over 600 education journals to display publicly-funded articles in ERIC 12 months after publication or sooner. If grantees or contractors publish their work in a participating journal, the journal will submit the full text to ERIC on behalf of the grantee so that the grantee will not need to submit their work to the ERIC Online Submission portal. Based on other Education responses to subsequent follow-up questions about these efforts, the enhancements to ERIC were completed in March 2020, and the number of Education-funded articles that were submitted through ERIC with links to research data grew from six as February 2021 to 21 as of April 2022.
Agency for Healthcare Research and Quality	The Director of the Agency for Healthcare Research and Quality should take steps to ensure appropriate agency-funded research data are readily findable and accessible to the public. (Recommendation 3)	Open – Partially Addressed The Agency for Healthcare Research and Quality concurred with this recommendation. According to an AHRQ update as of March 2022, the Data Management Plan (DMP) Policy that AHRQ released on May 26, 2020 was the first step toward making AHRQ funded research data available for public access. AHRQ stated that all Notices of Funding Opportunity Announcements and Requests for Proposals had been updated to include the DMP requirements, and that the DMP has become part of the terms and conditions of award, as appropriate, for grants and contracts. AHRQ's DMP policy directs researchers to describe their plan for the long term preservation of data generated from AHRQ funding and lists specific items that should be included in a DMP, such as how the data will be made discoverable, among other things. Additionally, AHRQ stated that it continues to participate in an interagency open science working group that is aimed at developing guidance and learning best practices to make federally funded research data available for public access. While we believe these efforts are steps to make AHRQ funded data more readily findable and accessible, in order for a member of the public to access AHRQ funded data, a request would need to be submitted to an agency contact person. In February 2023, an AHRQ official stated that, in response to the August 25, 2022, Office of Science and Technology Policy memorandum entitled "Ensuring Free, Immediate, and Equitable Access to Federally Funded Research," AHRQ submitted a draft updated public access plan on February 21, 2023. After receiving input on the revised public access plan, AHRQ will update its agency public access policy. Further the official said that, guided by its annual budget appropriation, AHRQ will consider all options for making data publicly available, including a web-based point of access that the public could use to find data. Once we determine the extent to which AHRQ has any additional plans to make AHRQ funded data more readily findable and accessible to the public, such as a single web-based point of access, we will provide an update.
Food and Drug Administration	The Commissioner of the Food and Drug Administration should take steps to ensure appropriate agency-funded research data are readily findable and accessible to the public. (Recommendation 4)	Open The Food and Drug Administration concurred with this recommendation. According to a June 2021 update, FDA planned to use a system called FindIT to catalog approved publications and datasets to make them readily findable and accessible to the public. After completing an initial pilot, a June 2022 FDA update stated that FindIT now includes a repository of datasets that have been approved for public access. FDA's planned next steps involve integrating the FindIT repository with FDA Center workflows for public dataset approval, according to the agency's update. In a June 2023 update, FDA stated it is still working to implement this recommendation. According to FDA's update, FDA stated that resource and funding issues have delayed the full implementation of FindIT and compliance monitoring, and a working group internal to FDA is being constituted with Center representation to directly address implementation of workflows and policy to address compliance and access. FDA further stated that OpenFDA will become the official repository of publicly accessible datasets, and FindIT will continue to be the "catalog" of what data exist with links to the data in OpenFDA. When we confirm additional steps taken by FDA to implement the recommendation, we will provide updated information.
Department of Homeland Security	The Secretary of Homeland Security should take steps to ensure appropriate agency-funded research data are readily findable and accessible to the public. (Recommendation 5)	Open – Partially Addressed The Department of Homeland Security (DHS) concurred with this recommendation and noted that it was in the process of establishing a portal on its website to increase public access to agency-funded research. In January 2022, DHS provided information demonstrating that a DHS-funded publications repository, hosted by NIH, has been linked to the DHS-funded data repository and is publicly available. According to DHS, metadata collected during posting of articles to the publications repository, including unique digital object identifiers, will be used to connect articles to the data in the data repository. As of April 2022, DHS said it continued to work toward making some additional improvements to this linkage, including reviewing the releasability of certain datasets. In March 2023, DHS provided an update stating that it created a website link on the main public facing DHS webpage that directs to an official DHS landing page that has descriptions and links to the various repositories for publicly funded research and associated data sets. A DHS official said this effort was complete as of December 2022. We requested documentation of this action and will update this recommendation when DHS provides additional information.
Department of Veterans Affairs	The Secretary of Veterans Affairs should take steps to ensure appropriate agency-funded research data are readily findable and accessible to the public. (Recommendation 6)	Open – Partially Addressed The Department of Veterans Affairs concurred with this recommendation but indicated in its comments on the report that it had already taken steps to implement it. As discussed in the report, the department's efforts to ensure research data availability pertained to a portion of the agency's federally-funded research data. In a June 11, 2020 response, VA officials outlined planned actions and timelines to address this recommendation. Specifically, VA officials stated they would discuss with agency leadership and other potential VA partners how to make a broader set of VA-funded data findable and accessible. They also said they would continue submitting a funding request to the VHA IT Committee for Operations and Maintenance to meet requirements for enabling data to be findable and accessible. In addition, they said they would work through the Veterans Informatics and Computing Infrastructure division to explore opportunities and methods for making data from other types of research publicly accessible. Subsequently, in December 2020, VA officials provided an update stating that the COVID-19 pandemic impacted these plans but that they still intend to pursue previously identified plans, along with new efforts such as increasing the use of VA Data Commons to allow access to clinical and genetic resources data. In February 2022, VA officials provided an update stating that its public access activities would be under the purview of a subcommittee it created for data governance, among other topics. They added that VA is still pursuing the creation of a VA Data Commons, which would include the ability to provide controlled access to research data for non-VA individuals. Furthermore, VA officials stated that it has plans to convene cognizant VA offices in late winter or early spring 2022 to develop a strategy to obtain funding to meet requirements for its proposed actions. In June 2022, VA provided an additional update in which it stated that its subcommittee had held initial meetings regarding its scope and role to advance enterprise approaches to managing ORD-funded research data. VA stated that the subcommittee would develop a written plan to outline further actions to implement the recommendation. In August 2023, VA officials met with GAO staff to provide a presentation on VA's efforts to implement this recommendation. The officials provided information on different pathways through which certain areas of VA-funded research data are made publicly available. Nevertheless, some gaps remain in the availability of VA-funded research data, and the officials said VA continues to pursue the effort to create VA Data Commons. We will update the status of this recommendation after we receive and evaluate any additional information on steps taken.
National Science Foundation	The Director of the National Science Foundation should fully implement plans to ensure appropriate agency-funded research data are readily findable and accessible to the public. (Recommendation 7)	Open – Partially Addressed NSF concurred with this recommendation. According to its response to our report, NSF is expanding its public access repository to include metadata records about data that support publications resulting from NSF-funded research. NSF's response stated that, by storing metadata records for supporting datasets alongside metadata records for publications, the public will be able to more easily find and access appropriate agency funded research data. In an NSF update as of March 2022, NSF stated that, in December 2021, it implemented NSF-PAR version 2.0 which enables the system to accept dataset records and ensure agency-funded research data are readily findable and accessible to the public. NSF provided evidence of dataset availability through the NSF-PAR system in March 2023 indicating that 577 datasets were available as of that time. NSF provided additional information in July 2023 describing the new features in NSF-PAR 2.5, such as enhanced search and filtering capabilities. According to the information NSF provided in July 2023, it is optional for researchers to add datasets to the PAR. NSF is working on developing guidance for researchers in advance of a January 31, 2025 effective date for agency requirements that researchers add datasets to the PAR. NSF expects to complete the guidance by December 31, 2024. We will provide an update when we obtain additional information about NSF's plans to require researchers to make datasets available through the NSF-PAR system.
Department of Defense	The Secretary of Defense should complete development of data management plan requirements for extramural researchers. (Recommendation 8)	Open – Partially Addressed The Department of Defense concurred with this recommendation stating data management plan (DMP) requirements would be in place by September 2023. In September 2022, DOD provided an update stating that it issued guidance in August 2021 requiring DMPs from grantees. DOD provided documentation of this guidance in May 2023, which states that a data management plan is required either as part of the application package prior to award or 30 days after award at the discretion of component program managers. DOD's September 2022 update also indicated that it had proposed DMP requirements for contractors into the DFARS. DOD provided a further update in June 2023 stating that the proposed changes to DFARS remain under review. We will provide an update once DOD provides additional information demonstrating the completion of these actions.
Agency for Healthcare Research and Quality	The Director of the Agency for Healthcare Research and Quality should complete development of data management plan requirements. (Recommendation 9)	Closed – Implemented In May 2020 the Agency for Healthcare Research and Quality released its final Data Management Plan Policy which established the agency's data management plan requirements. The policy requires applicants for AHRQ new or competing grants and research contracts to include a DMP for managing, storing and disseminating the primary data, samples, physical collections and other supporting materials created or gathered in the course of research funded by AHRQ, or state why data management is not possible, as a component of their grant application or research contract proposal. This policy applies to AHRQ intramural researchers and extramural recipients of AHRQ grants and research contracts, unless otherwise specified in the funding opportunity announcement or request for proposals
Department of Homeland Security	The Secretary of Homeland Security should complete development of data management plan requirements. (Recommendation 10)	Open – Partially Addressed The Department of Homeland Security (DHS) concurred with this recommendation, stating that it would develop a departmentwide management directive for research and development data as well as data management plan guidance and a template to document requirements. In a February 2021 update, DHS officials provided draft versions of its department-wide management directive for research data; a draft data management plan guidance document and template; and user guides relating to handling research data resulting from DHS funding. As of April 2022, DHS stated that it continues to work toward finalizing the directive. As of March 2023, DHS provided an estimated completion timeframe of late October 2023 for the directive. When we confirm the directive has been finalized, we will provide updated information.
U.S. Agency for International Development	The U.S. Agency for International Development Administrator should complete development of data management plan requirements for extramural researchers. (Recommendation 11)	Closed – Implemented The U.S. Agency for International Development agreed with this recommendation. In a June 2020 response USAID stated it would take action to establish data management plan (DMP) requirements, and was in the process of making substantial revisions to its data policy, found in Chapter 579 of its Automated Directives System (ADS), USAID Development Data. In September 2020, USAID officials responded that they completed the development of DMP requirements for extramural researchers, as applicable, revising Chapter 579 of its Development Data Policy to contain these new requirements. USAID officials added that they also established working groups to create additional technical-level guidance for DMPs. In March 2021, USAID officials confirmed that a July 31, 2020 revision of its policy was final, and the updated DMP requirements applied to extramural researchers.
Department of Agriculture	The Secretary of Agriculture should complete development of guidance and provide training to agency officials or others involved in reviewing the merits of researchers' data management plans. (Recommendation 12)	Closed – Implemented The U.S. Department of Agriculture agreed with the findings of our report. In October 2021, USDA provided a DMP review training plan that outlined a number of steps to develop training resources for DMP review and provide training to DMP reviewers. In February 2022, USDA provided additional information demonstrating that, as of September 2021, these training resources had been developed, and that USDA's Public Access Task Force had a plan in place to ensure that training is provided and training resources are available on an ongoing basis. As a result of these efforts, USDA will have better assurance that those responsible for reviewing the merits of researchers' DMPs are able to do so rigorously and in accordance with USDA's expectations for what DMPs should include.
U.S. Agency for International Development	The U.S. Agency for International Development Administrator should complete development of and provide training for agency officials or others involved in reviewing the merits of researchers' data management plans. (Recommendation 13)	Closed – Implemented The U.S. Agency for International Development agreed with this recommendation. In a June 2020 response, USAID stated it would complete the development of, and provide training for, agency officials or others involved in reviewing the merits of researchers' data management plans (DMPs). USAID noted it had already delivered preliminary training on data management planning and DMPs to specialized agency staff, and that it was advancing a training program that would include specific training and guidance on the development and evaluation of DMPs. In a September 2020 response, USAID officials provided an update on their training efforts, noting some disruptions due to COVID-19, but stating that training had resumed. They said training was provided to USAID's 12 Asia missions, focusing on best practices for data management planning, and was also developed for USAID's DATA Board on reviewing the merits of DMPs. USAID officials added that they had developed training guides and briefing documents to familiarize training participants with data management principles and necessary elements for a DMP. In a March 2021 update, USAID officials confirmed that its training efforts were offered for applicable staff that are responsible for reviewing the merits of researchers' data management plans, and that the training was being made available across all of USAID and its missions worldwide. They also said USAID plans to continue to offer aspects of its training for the foreseeable future.
Food and Drug Administration	The Commissioner of the Food and Drug Administration should evaluate training needs for agency officials or others involved in reviewing the merits of researchers' data management plans and, if additional training is found to be warranted, develop and provide such training. (Recommendation 14)	Open – Partially Addressed The Food and Drug Administration concurred with this recommendation. In June 2021, FDA provided an update stating that the FDA Library and the Office of Scientific Integrity were collaborating on agency-wide training, including public access training that incorporated requirements for both publications and data management plans. In June 2022, FDA provided an additional update stating that data management has been incorporated into a Responsible Conduct of Research course that is required for all researchers every 4 years. According to FDA's update, the data management section of the course discusses effective approaches to responsible data management and the ethical issues associated with data, including data collection, management, sharing, ownership, and protection. However, it was unclear from the information FDA provided whether this course is provided to individuals responsible for reviewing the merits of researchers' data management plans and the extent to which the course material is targeted to those individuals' needs. In June 2023, FDA provided an update, including data, on the number of FDA staff (researchers and supervisors) who had completed the course. Further, in August 2023, FDA provided documentation of course information that showed the inclusion of DMP requirements. At the same time, an FDA official said the agency does not require officials who review DMPs to take the course; although, in most cases, those officials will be researchers who have taken the training. The official also said the agency's public access policy provides criteria and elements that reviewers must consider in evaluating DMPs. As of September 2023, we are following up with FDA to obtain additional information on the potential implications of this training requirement gap for DMP reviewers. As we receive additional information from FDA, we will provide an update.
Department of Homeland Security	The Secretary of Homeland Security should evaluate training needs for agency officials or others involved in reviewing the merits of researchers' data management plans and, if additional training is found to be warranted, develop and provide such training. (Recommendation 15)	Open The Department of Homeland Security (DHS) concurred with this recommendation, and indicated it would evaluate training needs for data management plan (DMP) reviews and develop plans to fulfill any additional training needs identified. In a February 2021 update, DHS officials stated that training has commenced. According to an April 2022 update, DHS is planning a number of steps to provide training resources concurrent with the finalization of its directive. As of March 2023, DHS provided an estimated completion timeframe of late October 2023 for the directive. We will update this recommendation when DHS provides additional information.
Department of Veterans Affairs	The Secretary of Veterans Affairs should evaluate training needs for agency officials or others involved in reviewing the merits of researchers' data management plans and, if additional training is found to be warranted, develop and provide such training. (Recommendation 16)	Closed – Implemented The Department of Veterans Affairs (VA) concurred with this recommendation and identified several planned steps to identify and meet training needs for those involved in reviewing researchers' data management plans (DMPs). In a June 11, 2020 update, VA officials noted that it had completed a review of training needs in November 2019, which involved assessing its scientific review personnel's understanding of VA policies and standards related to DMPs. Through this process VA shared information on review processes, standards, and guidance from other agencies, in particular the National Institutes of Health and the National Institute for Standards and Technology, with its scientific personnel responsible for receipt, review and management of DMPs as part of their scope of activities. In April 2022, VA provided additional information on DMP-related training for Office of Research and Development staff that is part of VA's overall training process for handling research funding applications. This information included a November 2021 checklist that reviewers of DMPs use in evaluating applications, and that is now part of standard operating procedures and is included as part of training for new reviewers.
Agency for Healthcare Research and Quality	The Director of the Agency for Healthcare Research and Quality should evaluate training needs for agency officials or others involved in reviewing the merits of researchers' data management plans and, if additional training is found to be warranted, develop and provide such training. (Recommendation 17)	Closed – Implemented The Agency for Healthcare Research and Quality concurred with this recommendation stating that it would assess training needs for agency officials and others involved in reviewing the merits of researchers' data management plans and would develop and provide additional training if warranted. In response, AHRQ developed and provided documentation of training materials for scientific review groups and program officials. AHRQ also developed a review guide and checklists to assist the identification and assessment of data management plans. AHRQ provided training using these materials over a series of dates from March 2021 through January 2022. As a result, AHRQ will have better assurance that the merits of researchers' DMPs are being consistently evaluated.
Department of Defense	The Secretary of Defense should evaluate training needs for agency officials or others involved in reviewing the merits of researchers' data management plans and, if additional training is found to be warranted, develop and provide such training. (Recommendation 18)	Closed – Implemented The Department of Defense concurred with this recommendation. In a September 2022 update, DOD indicated it had completed several actions to implement the recommendation. First, according to DOD's update, it developed an automated data management plan tool for researchers to use at the beginning of a project, which was completed in September 2020. Second, DOD stated it created evaluation criteria training, which it deployed in November 2021. Third, DOD completed providing training in June 2022. DOD provided documentation of these actions in May 2023 and, as a result, we are considering this recommendation implemented.
Department of Energy	The Secretary of Energy should evaluate training needs for agency officials or others involved in reviewing the merits of researchers' data management plans and, if additional training is found to be warranted, develop and provide such training. (Recommendation 19)	Closed – Implemented The Department of Energy concurred with this recommendation and stated it would assess and develop a plan to meet the training needs of internal DOE staff and external peer reviewers of data management plans (DMPs). In October 2021, DOE 's Office of Science issued guidance for reviewers of DMPs, which included information on required DMP elements that reviewers should assess when evaluating research proposals. The guidance went into effect on January 1, 2022. Additionally, DOE updated the Office of Science Portfolio Analysis and Management System (PAMS), a web-based system for managing proposals and their peer reviews, to inform reviewers about this guidance and verify they are familiar with it in advance of reviewing proposals. DOE provided documentation showing that, as of February 2022, the PAMS updates had been implemented.
Environmental Protection Agency	The Environmental Protection Agency Administrator should evaluate training needs for agency officials or others involved in reviewing the merits of researchers' data management plans and, if additional training is found to be warranted, develop and provide such training. (Recommendation 20)	Closed – Implemented The Environmental Protection Agency agreed with this recommendation stating that it planned to evaluate training needs for agency officials who review researchers' data management plans. According to information EPA provided in December 2020 and May 2021, the agency had developed checklists and other materials for EPA managers and grant managers to use when reviewing data management plans. EPA further stated in December 2021 that these materials had been disseminated EPA-wide through the intranet site. In June 2022, EPA also provided information showing that training has been provided during its Public Access Forum's quarterly meetings, including, for example, one that occurred in March 2022.
Department of Agriculture	The Secretary of Agriculture should develop and implement a mechanism to ensure researcher compliance with the public access plan and associated requirements. (Recommendation 21)	Closed – Implemented The U.S. Department of Agriculture agreed with the findings of our report. In October 2021, USDA provided information on mechanisms the agency planned to institute to ensure that researchers comply with public access requirements. These mechanisms included, for example, establishing resources to communicate requirements to researchers; and developing a Departmental Regulation, guidance, and a compliance dashboard. In April 2022, USDA provided documentation of how ARS and NIFA ensure compliance with public access requirements, as well as USDA's departmentwide policies on public access to publications and data. These documents represent promising steps toward implementing the recommendation. However, researcher compliance with the USDA data policy, in particular, was partially contingent on its implementation in USDA's Departmental Regulations. In July 2022, USDA finalized the Departmental Regulation, formally establishing public access requirements and outlining roles and responsibilities department-wide.
Department of Defense	The Secretary of Defense should develop and implement a mechanism to ensure researcher compliance with the public access plan and associated requirements. (Recommendation 22)	Open – Partially Addressed The Department of Defense concurred with this recommendation. According to DOD officials, in May 2020, DOD research organizations performed a self-assessment of their compliance with public access requirements. The self-assessment found that DOD medical organizations were not sending articles stemming from DOD-funded research to DOD's Defense Technical Information Center (DTIC) per DOD policy. While articles that were jointly funded with the National Institutes of Health (NIH) were being sent to NIH's repository, PubMed Central, articles that were not jointly funded with NIH were not being made publicly available. To address this, DOD is taking steps to coordinate with NIH to ensure DOD-funded articles are made available through DOD's public facing interface, PubDefense, by developing a web service to download DOD-funded materials from PMC and also discussing with NIH whether non-NIH-funded documents may be deposited in PMC and then downloaded to DTIC's repository. Additionally, DOD officials stated the self-assessment led to an increase in submissions to DTIC and started conversations with DTIC on the best way to submit documents and DMPs to DTIC in an automated way. In a September 2022 update, DOD stated that it had established a methodology to determine noncompliance and identify the number of journal articles and DMPs generated but not sent to the DOD public access repository. In February 2023, DOD issued a memo directing compilation of an annual inventory of published journal articles, manuscripts, and supporting digital data to ensure researcher compliance. According to a DOD update, the initial FY22 inventory was due in June 2023. In October 2023, DOD provided an update stating that it expects to complete the analysis of the FY22 inventory in the second quarter of FY24. We will update the status of this recommendation when DOD provides additional information on these actions.
Agency for Healthcare Research and Quality	The Director of the Agency for Healthcare Research and Quality should develop and implement a mechanism to ensure researcher compliance with the public access plan and associated requirements.(Recommendation 23)	Closed – Implemented In May 2020, the Agency for Healthcare Research and Quality stated they have taken multiple steps to ensure compliance with the agency's public access plan and associated requirements including: (1) updating all Funding Opportunity Announcements and Requests for Proposals issued after May 26, 2020 to include the agency's Data Management Plan policy requirements; (2) training scientific review groups to identify and assess the Data Management Plan to determine if the plan is adequate; and (3) training all AHRQ staff, including Program Officials, Scientific Review Officers, and grants management staff on the Data Management Plan policy. In addition, AHRQ stated they also ensure compliance by including requirements in their Notice of Award terms and conditions for grants, cooperative agreements, and contracts and that failure to comply with the Terms and Conditions may result in an enforcement action, including additional special terms and conditions or termination of the award, and may affect future funding decisions. Additionally, during the funding period, AHRQ stated that their Program Officials continually assess compliance with the agency's public access plan and associated requirements. Compliance with the agency's public access plan and associated requirements may also be reviewed during regular reporting intervals, such as annual Research Performance Progress Reports (RPPRs).
Food and Drug Administration	The Commissioner of the Food and Drug Administration should develop and implement a mechanism to ensure researcher compliance with the public access plan and associated requirements. (Recommendation 24)	Open The Food and Drug Administration concurred with this recommendation. FDA planned to implement the recommendation by using its FDA Library FindIT system to ascertain and report researcher compliance with the public access plan and associated requirements, according to a June 2021 update. In June 2022, FDA provided an additional update in which it stated that FindIT is collecting information on publications authored by FDA. According to FDA, this information will be compared with deposits in the PubMed Central system to report compliance. In a June 2023 update, FDA stated it is still working to implement this recommendation. According to FDA's update, the Office of Science and Technology Policy has formed several committees and working groups that are focused on refining the government-wide approach to making the results of federally funded research available to the public. FDA has representation on several of those working groups and committees, and it is following the work that is being done at that level to develop standards for monitoring compliance. FDA stated that resource and funding issues have delayed the full implementation of FindIT and compliance reporting, and a working group internal to FDA is being constituted with Center representation to directly address implementation of workflows and policy to address compliance and access. When we confirm additional steps taken by FDA to implement the recommendation, including how the FindIT system is being used to ensure compliance with both publications and data public access requirements, we will provide updated information.
National Institutes of Health	The Director of the National Institutes of Health should fully develop and implement a mechanism to ensure researcher compliance with the public access plan and associated requirements. (Recommendation 25)	Closed – Implemented In October 2020, NIH issued its Data Management and Sharing Policy and accompanying supplemental information. According to the policy, it applies to all research, funded or conducted in whole or in part by NIH, that generates scientific data. The effective date of the NIH policy is January 25, 2023 (i.e., for competitive grant applications or contract proposals submitted for that date, or other research projects conducted on or after that date). The policy states that compliance with the data management and sharing plans developed pursuant to it is a term and condition of extramural awards and contracts, and that compliance for intramural research projects and other funding agreements will be enforced consistent with applicable NIH policies. The policy also indicates that, after the end of a funding period, non-compliance with a data management and sharing plan can be considered by NIH in determining whether to provide future funding to recipient institutions.
Department of Homeland Security	The Secretary of Homeland Security should develop and implement a mechanism to ensure researcher compliance with the public access plan and associated requirements. (Recommendation 26)	Open The Department of Homeland Security (DHS) concurred with this recommendation and stated it would develop a mechanism to ensure researcher compliance with the department's public access plan and data management plan requirements. According to a September 2021 update, DHS said that its components that conduct R&D provide reviewers that serve as administrative gatekeepers. DHS said it expected that usage rates for publications and data repositories would increase since mandatory metadata must be entered before articles and data can be placed in a repository. Both repositories have the ability to gather user metrics that DHS will use to determine user compliance. DHS said it will use that data to adjust practices and procedures to improve compliance, as necessary. In a January 2022 update, DHS clarified that the administrative gatekeepers are responsible for ensuring that publications and data are made publicly available to the extent appropriate. Additionally, in a March 2023 update DHS said it was reviewing and revising the Department's Public Access Plan (last published in 2016) based on the OSTP August 2022 memo. As part of the revision process, DHS will include a mechanism to ensure researcher compliance with the Public Access Plan, as well as the new DMP Directive & Instruction. DHS indicated an estimated completion date of late October 2023. We will provide an update when DHS provides additional information and documentation on these actions.
National Oceanic and Atmospheric Administration	The National Oceanic and Atmospheric Administration Administrator should fully develop and implement a mechanism to ensure researcher compliance with the public access plan and associated requirements. (Recommendation 27)	Open The Department of Commerce agreed with this recommendation and the National Oceanic and Atmospheric Administration indicated it was pursuing multiple mechanisms to implement it. According to information NOAA provided in January and June 2021, the agency continues to pursue a multi-pronged approach to better understand the extent of compliance issues with its public access plan and to address them. For instance, in May 2021, NOAA's Acting Chief Scientist communicated the importance of complying with the public access plan agency-wide. NOAA's June 2021 update also indicated that the agency's Science Council continues to investigate the reasons for, and proposed changes to improve public access plan compliance. GAO met with NOAA officials in March 2022 to discuss implementation of this recommendation and NOAA agreed to follow up with additional information. As we obtain more information about NOAA's actions in response to this recommendation, we will provide updated information.
Department of Energy	The Secretary of Energy should fully develop and implement a mechanism to ensure researcher compliance with the public access plan and associated requirements. (Recommendation 28)	Closed – Implemented The Department of Energy concurred with this recommendation stating that it would develop a compliance mechanism to identify researchers receiving funding from DOE financial assistance awards who are not compliant with DOE's public access plan for publications. According to information DOE provided, DOE's Office of Scientific and Technical Information (OSTI) engaged individual DOE programs/awarding offices to discuss plans and approaches for ensuring researcher compliance with public access and to determine a primary compliance mechanism that may be used to identify accepted manuscripts published by DOE financial assistance awardees (extramural researchers). DOE's OSTI developed compliance reports for the DOE program offices that fund extramural scientific research, which DOE began using in January 2021, according to an update DOE provided in March 2022. DOE stated that these compliance reports will be used to monitor and report compliance to the DOE programs/awarding offices to inform them where gaps in compliance exist for their funded researchers, and that this compliance mechanism will assist the DOE programs in understanding how their awardees are progressing in terms of compliance with public access.
Department of Veterans Affairs	The Secretary of Veterans Affairs should fully develop and implement a mechanism to ensure researcher compliance with the public access plan and associated requirements. (Recommendation 29)	Open The Department of Veterans Affairs concurred with this recommendation but indicated in its comments on the report that it had already taken steps to implement it. As discussed in the report, the department's public access compliance mechanism covered a portion of the agency's federally-funded research data. In a June 2020 update, VA indicated it planned to examine opportunities and develop plans that address public access for other types of research it supports, including compliance mechanisms. In a December 2020 update, VA officials noted that the COVID-19 pandemic had resulted in delays in its efforts. In an August 2021 update, VA officials said actions for researcher compliance are taken prior to submission of a funding request, and are further enforced as a condition of funding, although VA officials did not elaborate on the compliance mechanisms, nor did they provide documents to substantiate the statements. In February 2022 and June 2022 updates as well as during an August 2023 meeting, VA officials said that key updates and outcomes related to addressing this recommendation are linked to VA's pursuit of a strategy making research data available through a data sharing infrastructure. When we confirm what actions the agency has taken in response to the recommendation, we will provide another update.
Department of Transportation	The Secretary of Transportation should fully develop and implement a mechanism to ensure researcher compliance with the public access plan and associated requirements. (Recommendation 30)	Open The Department of Transportation concurred with this recommendation. According to its response to our report, the Department will build upon existing compliance mechanisms to ensure researcher compliance with its public access plan and associated requirements. As part of this process, the Department reported that it plans to update its public access plan, and expects to complete these actions by December 31, 2020. On July 22, 2021, Transportation officials provided an update stating that the Department is working toward implementation of the identified actions by January 15, 2022. As of August 2023, we have requested an update from the Department. When we confirm what actions the Department has taken to implement this recommendation we will provide additional information.
National Institute of Standards and Technology	The National Institute of Standards and Technology Director should fully develop and implement a mechanism to ensure researcher compliance with the public access plan and associated requirements. (Recommendation 31)	Open – Partially Addressed The Department of Commerce agreed with this recommendation and the National Institute of Standards and Technology (NIST) identified several planned steps to implement it. Specifically, in January 2021, NIST deployed a system developed to identify NIST-authored papers that have been published, and ensure that all NIST authors are in compliance with the requirement that papers are made freely available within 12 months of publication. Previously, NIST also completed a review of awardees' compliance with data management plan requirements, and took steps (e.g., developing a template and checklist) to address issues NIST identified. Additionally, NIST took steps to evaluate an option to ensure compliance with public access requirements by NIST awardees (i.e., extramural researchers). NIST determined that the option it evaluated would not be practical as of April 2021. As of August 2023, NIST stated that it is not yet able to report awardees' compliance with the terms and conditions in their grants because the agency's current Grants Management Information System (GMIS) does not track that. According to NIST's update, it is transitioning from GMIS to a different system (eRA Commons), which will be implemented in October 2024. NIST anticipates that the new system will allow monitoring of extramural compliance and the agency should be able to implement compliance tracking in December 2024. We will further update the recommendation status when NIST provides additional information on steps taken to ensure compliance with public access requirements by NIST's extramural researchers.
Office of Science and Technology Policy	Priority Rec. As the Subcommittee on Open Science moves forward, the Office of Science and Technology Policy co-chair, in coordination with other co-chairs and participating agencies, should take steps to fully implement leading practices that enhance and sustain collaboration. (Recommendation 32)	Open – Partially Addressed OSTP initially disagreed with GAO's November 2019 recommendation, stating that the subcommittee had already taken steps to implement the leading practices GAO identified; however, OSTP officials did not provide documentation of these efforts and GAO continues to believe the recommendation is warranted. In January 2022, OSTP provided an update on information it previously provided in October 2020 and May 2021 on steps the subcommittee has taken to address issues associated with public access to federally funded research results, including certain areas GAO identified as presenting challenges to public access plan implementation in the November 2019 report. In August 2022, the subcommittee released a new charter that outlines its functions, its member agencies including co-chairs, and identifies certain activities that may be needed to implement its functions such as convening meetings and gathering input from stakeholders. However, the charter does not fully address the three leading practices GAO evaluated in the 2019 report. For example, the charter does not discuss how the results of the subcommittee's collaborative efforts will be monitored, evaluated, and reported. In February 2023, OSTP provided additional information on the subcommittee's efforts including identifying the focus areas of the subcommittee's six subgroups and detailing the subcommittee's current agency members. However, it is not yet clear from the information OSTP provided how the subcommittee is fully implementing certain practices GAO evaluated, such as defining and articulating common outcomes and developing mechanisms to monitor, evaluate, and report on results. We will continue to collect and evaluate additional information to determine the extent to which the subcommittee's work processes incorporate leading practices for interagency collaboration that GAO has identified. By taking steps to fully implement the relevant leading practices GAO has identified, the subcommittee and its member agencies could better marshal their collective efforts to support public access to research results.
Department of Defense	As the Subcommittee on Open Science moves forward, the Department of Defense co-chair, in coordination with other co-chairs and participating agencies, should take steps to fully implement leading practices that enhance and sustain collaboration. (Recommendation 33)	Open – Partially Addressed DOD concurred with GAO's November 2019 recommendation. As of May 2021, DOD provided information on steps it had taken or was taking as a result of its participation in the subcommittee to implement the recommendation. The information DOD previously provided generally described steps the subcommittee had taken to address issues associated with public access to federally funded research results, including certain areas GAO identified as presenting challenges to public access plan implementation in the November 2019 report. In August 2022, the National Science and Technology Council's Committee on Science re-established the charter for the Subcommittee on Open Science. Under the new charter, DOD is no longer listed as a co-chair of the subcommittee. We received an update from the Office of Science and Technology Policy in February 2023 on efforts to implement this recommendation. We will continue to follow up with the Office of Science and Technology Policy and other co-chairing agencies to obtain information on the extent to which the subcommittee is implementing leading practices that enhance and sustain collaboration.
Department of Energy	As the Subcommittee on Open Science moves forward, the Department of Energy co-chair, in coordination with other co-chairs and participating agencies, should take steps to fully implement leading practices that enhance and sustain collaboration. (Recommendation 34)	Open – Partially Addressed DOE concurred with GAO's November 2019 recommendation. As of January 2022, DOE had provided information on steps taken to implement the recommendation. The information DOE provided generally described steps the subcommittee has taken to address issues associated with public access to federally funded research results, including certain areas GAO identified as presenting challenges to public access plan implementation in the November 2019 report. In August 2022, the National Science and Technology Council's Committee on Science re-established the charter for the Subcommittee on Open Science. Under the new charter, DOE is no longer listed as a co-chair of the subcommittee. We received an update from the Office of Science and Technology Policy in February 2023 on efforts to implement this recommendation. We will continue to follow up with the Office of Science and Technology Policy and other co-chairing agencies to obtain information on the extent to which the subcommittee is implementing leading practices that enhance and sustain collaboration.
National Institutes of Health	As the Subcommittee on Open Science moves forward, the National Institutes of Health co-chair, in coordination with other co-chairs and participating agencies, should take steps to fully implement leading practices that enhance and sustain collaboration. (Recommendation 35)	Open – Partially Addressed NIH concurred with GAO's November 2019 recommendation and has taken steps to implement it. According to information NIH provided in December 2021, the subcommittee is continuing to implement interagency efforts and to develop evidence demonstrating how the processes of the subcommittee implement the leading practices that GAO has identified can enhance and sustain interagency collaboration. In September 2022, NIH provided an update in which it stated that the subcommittee has established a robust and collaborative partnership with its interagency partners and external stakeholders. NIH provided recent work products of the Subcommittee including a guidance document on Desirable Characteristics of Data Repositories for Federally Funded Research and an August 2022 OSTP memorandum providing guidance and recommendations for agencies in implementing their public access policies. However, the information provided generally focused on the work products of the subcommittee, rather than on how the leading practices have been incorporated into the work processes of the subcommittee. Also, in August 2022, the subcommittee released a new charter that outlines its functions, its member agencies including co-chairs, and identifies certain activities that may be needed to implement its functions such as convening meetings and gathering input from stakeholders. However, the charter does not fully address the three leading practices GAO evaluated in the 2019 report. For example, the charter does not discuss how the results of the subcommittee's collaborative efforts will be monitored, evaluated, and reported. In February 2023, NIH and OSTP provided additional information on the subcommittee's efforts including identifying the focus areas of the subcommittee's six subgroups and detailing the subcommittee's current agency members. However, it is not yet clear from the information provided how the subcommittee is fully implementing certain practices GAO evaluated, such as defining and articulating common outcomes and developing mechanisms to monitor, evaluate, and report on results. We will continue to collect and evaluate additional information to determine the extent to which the subcommittee's work processes incorporate leading practices for interagency collaboration that GAO has identified. By taking steps to fully implement the relevant leading practices GAO has identified, the subcommittee and its member agencies could better marshal their collective efforts to address common public access plan implementation challenges that agency officials and stakeholders identified.
National Oceanic and Atmospheric Administration	As the Subcommittee on Open Science moves forward, the National Oceanic and Atmospheric Administration co-chair, in coordination with other co-chairs and participating agencies, should take steps to fully implement leading practices that enhance and sustain collaboration. (Recommendation 36)	Open – Partially Addressed NOAA concurred with GAO's November 2019 recommendation, noting that NOAA would work with the subcommittee to identify more opportunities for collaboration to promote access to research results. NOAA provided information in August 2020 and June 2021 on efforts to implement the recommendation. The information NOAA provided generally described steps taken to work with other agencies to address issues associated with public access to federally funded research results, including certain areas GAO identified as presenting challenges to public access plan implementation in the November 2019 report. In August 2022, the National Science and Technology Council's Committee on Science re-established the charter for the Subcommittee on Open Science. Under the new charter, NOAA is no longer listed as a co-chair of the subcommittee. We received an update from the Office of Science and Technology Policy in February 2023 on efforts to implement this recommendation. We will continue to follow up with the Office of Science and Technology Policy and other co-chairing agencies to obtain information on the extent to which the subcommittee is implementing leading practices that enhance and sustain collaboration.
National Science Foundation	As the Subcommittee on Open Science moves forward, the National Science Foundation co-chair, in coordination with other co-chairs and participating agencies, should take steps to fully implement leading practices that enhance and sustain collaboration. (Recommendation 37)	Open – Partially Addressed NSF concurred with GAO's November 2019 recommendation and has taken steps to implement it. According to information NSF provided in December 2021, the subcommittee was undertaking interagency efforts to address issues associated with public access to federally funded research results, including certain areas GAO identified as presenting challenges to public access plan implementation in the November 2019 report. In August 2022, the subcommittee released a new charter that outlines its functions, its member agencies including co-chairs, and identifies certain activities that may be needed to implement its functions such as convening meetings and gathering input from stakeholders. However, the charter does not fully address the three leading practices GAO evaluated in the 2019 report. For example, the charter does not discuss how the results of the subcommittee's collaborative efforts will be monitored, evaluated, and reported. In February 2023, OSTP provided additional information on the subcommittee's efforts including identifying the focus areas of the subcommittee's six subgroups and detailing the subcommittee's current agency members. However, it is not yet clear from the information provided how the subcommittee is fully implementing certain practices GAO evaluated, such as defining and articulating common outcomes and developing mechanisms to monitor, evaluate, and report on results. Further, in March 2023, NSF provided an update indicating that the subcommittee's efforts are guided by a workplan that includes linkages between actions and deliverables but did not provide documentation of the workplan. GAO will continue to collect and evaluate additional information to determine the extent to which these steps incorporate leading practices for interagency collaboration GAO has identified. By taking steps to fully implement the relevant leading practices GAO has identified, the subcommittee and its member agencies could better marshal their collective efforts to address common public access plan implementation challenges that agency officials and stakeholders identified.

See All 37 Recommendations

Full Report

Highlights Page (1 page)

Full Report (96 pages)

Accessible PDF (143 pages)

GAO Contacts

John Neumann

Managing Director

neumannj@gao.gov

(202) 512-6888

Office of Public Affairs

Chuck Young

Managing Director

youngc1@gao.gov

(202) 512-4800

Topics

Science and Technology

Federally funded researchScience and technologyHealth careConservationFoodResearch and development expendituresPublicationsResearch and developmentInteragency relationsFederal agencies