Peer Review: Reforms Needed to Ensure Fairness in Federal Agency Grant Selection
PEMD-94-1
Published: Jun 24, 1994. Publicly Released: Jul 26, 1994.
Skip to Highlights
Highlights
Pursuant to a congressional request, GAO reviewed the fairness of peer review processes for awarding federal grants at three federal agencies, focusing on: (1) potential peer review weaknesses; (2) agency policies to address these potential weaknesses; and (3) reviewer selection, conduct, and grant award decisions.
Recommendations
Recommendations for Executive Action
| Agency Affected | Recommendation | Status |
|---|---|---|
| National Institutes of Health | The Directors of the National Institutes of Health (NIH) and National Science Foundation (NSF), and the Chairman, National Endowment for the Humanities (NEH), should address the underrepresentation of young, junior scholars as reviewers by targeted outreach efforts, at least experimentally. |
Closed – Implemented
NIH established a database of potential reviewers that includes a higher proportion of junior faculty than are currently serving on review panels. Review staff are being actively encouraged to make increasing use of this database in selecting reviewers, regardless of age or academic rank.
|
| National Endowment for the Humanities | The Directors of the National Institutes of Health (NIH) and National Science Foundation (NSF), and the Chairman, National Endowment for the Humanities (NEH), should address the underrepresentation of young, junior scholars as reviewers by targeted outreach efforts, at least experimentally. |
Closed – Implemented
NEH says it has increased the number of junior scholars in its database through a variety of means. NEH has found that young scholars "are often thorough and energetic reviewers and panelists."
|
| National Science Foundation | The Directors of the National Institutes of Health (NIH) and National Science Foundation (NSF), and the Chairman, National Endowment for the Humanities (NEH), should address the underrepresentation of young, junior scholars as reviewers by targeted outreach efforts, at least experimentally. |
Closed – Not Implemented
Although GAO presented evidence to the contrary, NSF believes its ongoing outreach efforts to be sufficient.
|
| National Institutes of Health | The Directors, NIH and NSF, and the Chairman, NEH, should increase the monitoring of discrimination, including conducting tests comparing blind to conventional reviews, to ensure that gender, race, and ethnic discrimination are not affecting scores given by peer reviewers. |
Closed – Not Implemented
NIH disagrees with the recommendation and instead is focusing on closer real time monitoring of discrimination in scoring, encouraging review staff to aggressively challenge or even remove scores that appear outside the range of other reviewers.
|
| National Endowment for the Humanities | The Directors, NIH and NSF, and the Chairman, NEH, should increase the monitoring of discrimination, including conducting tests comparing blind to conventional reviews, to ensure that gender, race, and ethnic discrimination are not affecting scores given by peer reviewers. |
Closed – Implemented
NEH planned to do a blind test, but the program was cut when its appropriation was reduced.
|
| National Science Foundation | The Directors, NIH and NSF, and the Chairman, NEH, should increase the monitoring of discrimination, including conducting tests comparing blind to conventional reviews, to ensure that gender, race, and ethnic discrimination are not affecting scores given by peer reviewers. |
Closed – Not Implemented
NSF disagreed with the GAO finding, citing data on funding rates rather than direct evidence of review scores. It has not conducted the studies GAO recommended to resolve these discrepancies, saying that "such a test does not have a high priority".
|
| National Institutes of Health | The Directors, NIH and NSF, and the Chairman, NEH, should address the lack of clarity in the application of review criteria by developing rating systems in which proposals are rated separately on a number of criteria (such as importance of the issue, quality of the design, institutional support, and qualifications of the applicant), along with the overall rating. |
Closed – Implemented
NIH established a Rating Grant Applications Committee to address this issue. It concurred with the GAO recommendation to rate, or score, the individual criteria separately. However, others in the health research community disagreed. NIH has developed new criteria to be used for the review of all grant applications, beginning with applications reviewed for the January 1998 councils. Reviewers will be instructed to address and evaluate the five review criteria independently. However, they will only assign a single, global score for each application.
|
| National Endowment for the Humanities | The Directors, NIH and NSF, and the Chairman, NEH, should address the lack of clarity in the application of review criteria by developing rating systems in which proposals are rated separately on a number of criteria (such as importance of the issue, quality of the design, institutional support, and qualifications of the applicant), along with the overall rating. |
Closed – Implemented
A task force considered how to effectively implement. It recommended to division directors that instructions to panelists be made more explicit, that criteria be numbered, and that panelists be asked to address each criterion.
|
| National Science Foundation | The Directors, NIH and NSF, and the Chairman, NEH, should address the lack of clarity in the application of review criteria by developing rating systems in which proposals are rated separately on a number of criteria (such as importance of the issue, quality of the design, institutional support, and qualifications of the applicant), along with the overall rating. |
Closed – Implemented
NSF clarified and simplified its criteria. Reviewers are now asked to address each criterion as it applies to the particular proposal, and to provide an overall rating with a brief summary statement that includes comments on the relative importance of the two criteria in the reviewer's rating. However, it has, after considerable debate, decided not to have a numerical rating, or scoring, of each criterion.
|
| National Institutes of Health | The Directors, NIH and NSF, and the Chairman, NEH, should identify any commonly used unwritten or informal decision rules that are applied by peer reviewers and, where feasible, formalize them or at least inform applicants of their importance. |
Closed – Implemented
PHS' 398 grant application kit is being revised and applicants will be instructed to include information on preliminary studies, and a tear-out request form for further information is being added. The Rating Grant Applications Committee is further reviewing this issue.
|
| National Endowment for the Humanities | The Directors, NIH and NSF, and the Chairman, NEH, should identify any commonly used unwritten or informal decision rules that are applied by peer reviewers and, where feasible, formalize them or at least inform applicants of their importance. |
Closed – Implemented
Panelists will continue to be instructed to maintain their discussion within the scope of the criteria.
|
| National Science Foundation | The Directors, NIH and NSF, and the Chairman, NEH, should identify any commonly used unwritten or informal decision rules that are applied by peer reviewers and, where feasible, formalize them or at least inform applicants of their importance. |
Closed – Implemented
NSF simplified and clarified its criteria. "The system will be improved only if the reviewers use the criteria when evaluating the proposal. Thus...criteria...must be formatted for maximal use." Guidance to reviewers now also asks reviewers to comment on the relative importance they gave to the two principal criteria. Guidance to staff asks that additional program-specific criteria be included in the announcement. Staff are also asked to mention the new criteria in any correspondence with the proposing communities. However, what is not clear is the extent to which NSF first identified any commonly used unwritten or informal decision rules before developing the new criteria.
|
| National Institutes of Health | The Director, NIH, should produce a better match of panelists' area of expertise to that of the proposals by making greater use of subpanels and more fully integrating the work of ad hoc mail reviewers into the panel process. |
Closed – Implemented
NIH is making somewhat greater use of subpanels. It is allowing ad hoc reviewers to vote in some circumstances and has changed its guidance to SRAs to distribute mail reviews to more members of the panel.
|
| National Institutes of Health | The Director, NIH, should improve evaluation and oversight by retaining data on scores given by individual panelists and the race and gender of individual applicants. |
Closed – Not Implemented
NIH disagrees with the recommendation, arguing that the same results can be achieved through its program of monitoring for discrimination. However, by destroying the data, NIH makes full congressional oversight impossible.
|
| National Science Foundation | The Director, NSF, should address the potential for bias arising out of extensive personal relationships between reviewers and applicants by increasing the use of panels, where feasible. |
Closed – Implemented
The percentage of NSF reviews done solely by mail review has decreased, while the proportion of reviews done by both panels and mail review has increased.
|
| National Science Foundation | The Director, NSF, should address reviewer representativeness by more closely monitoring the inclusion of women and minorities among external reviewers. |
Closed – Implemented
NSF is planning to collect some limited data on the inclusion of women and minorities among their external reviewers, in some directorates but not all. However, it is a very limited effort. Unfortunately, as NSF officials GAO interviewed pointed out, the reason the representativeness of the internal or panel reviews had improved while that of the external reviewers had not, was because it was closely monitoring the composition of panels but not of the external, mail reviewers. This is particularly important to those fields relying heavily or exclusively on external review.
|
| National Science Foundation | The Director, NSF, should increase efforts to calibrate proposal ratings among reviewers through information provided in advance and discussions of examples at the convening of panels. |
Closed – Implemented
Action has been completed.
|
| National Endowment for the Humanities | The Chairman, NEH, should improve the level of relevant expertise in reviews by making greater use of mail reviewers. |
Closed – Not Implemented
A task force examined mail review and recommended that the different programs continue to use the type of review "that suits their needs." There is no plan to increase the use of mail reviewers, but rather to simply improve the technology used to communicate those reviews.
|
| National Endowment for the Humanities | The Chairman, NEH, should improve evaluation and oversight by retaining data on the race of applicants. |
Closed – Implemented
Beginning in FY 1995, NEH has requested and retained data on race/ethnicity and gender of applicants and reviewers. The task force recommended that the data be kept separately from all other records.
|
| National Endowment for the Humanities | The Chairman, NEH, should increase efforts to calibrate proposal ratings among reviewers through information provided in advance and discussions of examples at the convening of panels. |
Closed – Implemented
A task force suggested that panelists get more explicit instruction and more opportunity for the early calibration of their assessments.
|
Full Report
GAO Contacts
Office of Public Affairs
Topics
Conflict of interestsEvaluation methodsFederal aid for the humanitiesGrant award proceduresResearch grantsPeer reviewGraphic user interfaceFederal researchMass mediaFederal agencies