Peer Review Practices at Federal Science Agencies Vary
RCED-99-99: Published: Mar 17, 1999. Publicly Released: Apr 19, 1999.
Pursuant to a congressional request, GAO studied the peer review and other quality assurance processes that federal agencies use in conducting scientific research and development, focusing on: (1) defining what is meant by peer review; (2) describing the federal government's peer review policy; (3) describing the peer review practices of 12 federal agencies that conduct scientific research; (4) describing other agencies' quality assurance reviews; and (5) identifying which research is not subject to review.
GAO noted that: (1) there is no written governmentwide definition of peer review; (2) officials at the Office of Science and Technology Policy (OSTP) and at the agencies GAO contacted generally concur that peer review is defined as a process that includes an independent assessment of the technical, or scientific merit of research by peers who are scientists with knowledge and expertise equal to that of the researchers whose work they review; (3) there is no uniform federal policy for conducting peer reviews; (4) through annual budget guidance to federal agencies, OSTP and the Office of Management and Budget encourage funding of research projects that are peer reviewed over those that are not reviewed; (5) officials at OSTP said that peer review practices should not be dictated uniformly for every agency or for all types of federally funded research; (6) rather, the practices should be tailored to agency missions and type of research; (7) each of the 12 agencies that GAO contacted had a variety of policies, orders, or other internal guidance regarding the conduct of peer review; (8) to varying degrees, the 12 agencies use peer review to: (a) assess the merit of competitive and noncompetitive research proposals; (b) determine whether to continue or renew research projects; (c) evaluate the results of the research prior to the publication of those results; (d) establish annual budget priorities for research programs; and (e) evaluate program and scientist performance; (9) all of the agencies use peer review to assess competitive research proposals; (10) the methods for conducting peer reviews vary among and within the agencies; (11) most of the agencies that GAO reviewed also use reviews by agency supervisors or program managers to assess the quality of research proposals, to check the quality of in-progress research, and to evaluate program performance; (12) generally, these quality assurance reviews are not considered independent assessments--a key criterion in the peer review process; (13) these quality assurance reviews occur at both the project and program levels; (14) while agencies reported that almost all research is reviewed either through peer reviews or other quality assurance reviews, a small amount of research may not be reviewed by the agencies in certain circumstances; and (15) examples of research that may be funded without being reviewed include projects that are congressionally mandated or projects that use widely accepted methodologies.