Peer Review:

Reforms Needed to Ensure Fairness in Federal Agency Grant Selection

PEMD-94-1: Published: Jun 24, 1994. Publicly Released: Jul 26, 1994.

Additional Materials:

Contact:

Nancy R. Kingsbury
(202) 512-7824
contact@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

Pursuant to a congressional request, GAO reviewed the fairness of peer review processes for awarding federal grants at three federal agencies, focusing on: (1) potential peer review weaknesses; (2) agency policies to address these potential weaknesses; and (3) reviewer selection, conduct, and grant award decisions.

GAO found that: (1) peer review processes appear to be working well and are generally supported by peer reviewers; (2) although the three agencies have policies to prevent financial and institutional conflicts of interest, they cannot rigorously screen for professional conflicts of interest; (3) the more expertise reviewers have in a given subject, the more they are likely to know the applicants personally; (4) the agencies need to improve reviewer selection in the areas of academic standing, gender, and expertise; (5) peer reviewers inconsistently apply agency review criteria and rating scales and tend to use unwritten rules in rating proposals; (6) proposals submitted by women and minorities receive significantly lower scores than those submitted by men and whites; (7) the agencies vary in the importance of peer review scores to their final award decisions; and (8) an applicant with a strong publication track record is more likely to receive funding.

Recommendations for Executive Action

  1. Status: Closed - Implemented

    Comments: A task force suggested that panelists get more explicit instruction and more opportunity for the early calibration of their assessments.

    Recommendation: The Chairman, NEH, should increase efforts to calibrate proposal ratings among reviewers through information provided in advance and discussions of examples at the convening of panels.

    Agency Affected: National Science Foundation

  2. Status: Closed - Implemented

    Comments: A task force considered how to effectively implement. It recommended to division directors that instructions to panelists be made more explicit, that criteria be numbered, and that panelists be asked to address each criterion.

    Recommendation: The Directors, NIH and NSF, and the Chairman, NEH, should address the lack of clarity in the application of review criteria by developing rating systems in which proposals are rated separately on a number of criteria (such as importance of the issue, quality of the design, institutional support, and qualifications of the applicant), along with the overall rating.

    Agency Affected: National Foundation on the Arts and Humanities: National Endowment for the Humanities

  3. Status: Closed - Implemented

    Comments: NIH established a Rating Grant Applications Committee to address this issue. It concurred with the GAO recommendation to rate, or score, the individual criteria separately. However, others in the health research community disagreed. NIH has developed new criteria to be used for the review of all grant applications, beginning with applications reviewed for the January 1998 councils. Reviewers will be instructed to address and evaluate the five review criteria independently. However, they will only assign a single, global score for each application.

    Recommendation: The Directors, NIH and NSF, and the Chairman, NEH, should address the lack of clarity in the application of review criteria by developing rating systems in which proposals are rated separately on a number of criteria (such as importance of the issue, quality of the design, institutional support, and qualifications of the applicant), along with the overall rating.

    Agency Affected: National Science Foundation

  4. Status: Closed - Implemented

    Comments: NSF clarified and simplified its criteria. Reviewers are now asked to address each criterion as it applies to the particular proposal, and to provide an overall rating with a brief summary statement that includes comments on the relative importance of the two criteria in the reviewer's rating. However, it has, after considerable debate, decided not to have a numerical rating, or scoring, of each criterion.

    Recommendation: The Directors, NIH and NSF, and the Chairman, NEH, should address the lack of clarity in the application of review criteria by developing rating systems in which proposals are rated separately on a number of criteria (such as importance of the issue, quality of the design, institutional support, and qualifications of the applicant), along with the overall rating.

    Agency Affected: National Science Foundation

  5. Status: Closed - Implemented

    Comments: NEH planned to do a blind test, but the program was cut when its appropriation was reduced.

    Recommendation: The Directors, NIH and NSF, and the Chairman, NEH, should increase the monitoring of discrimination, including conducting tests comparing blind to conventional reviews, to ensure that gender, race, and ethnic discrimination are not affecting scores given by peer reviewers.

    Agency Affected: National Science Foundation

  6. Status: Closed - Not Implemented

    Comments: NIH disagrees with the recommendation and instead is focusing on closer real time monitoring of discrimination in scoring, encouraging review staff to aggressively challenge or even remove scores that appear outside the range of other reviewers.

    Recommendation: The Directors, NIH and NSF, and the Chairman, NEH, should increase the monitoring of discrimination, including conducting tests comparing blind to conventional reviews, to ensure that gender, race, and ethnic discrimination are not affecting scores given by peer reviewers.

    Agency Affected: Department of Health and Human Services: Public Health Service: National Institutes of Health

  7. Status: Closed - Not Implemented

    Comments: NSF disagreed with the GAO finding, citing data on funding rates rather than direct evidence of review scores. It has not conducted the studies GAO recommended to resolve these discrepancies, saying that "such a test does not have a high priority".

    Recommendation: The Directors, NIH and NSF, and the Chairman, NEH, should increase the monitoring of discrimination, including conducting tests comparing blind to conventional reviews, to ensure that gender, race, and ethnic discrimination are not affecting scores given by peer reviewers.

    Agency Affected: Department of Health and Human Services: Public Health Service: National Institutes of Health

  8. Status: Closed - Not Implemented

    Comments: Although GAO presented evidence to the contrary, NSF believes its ongoing outreach efforts to be sufficient.

    Recommendation: The Directors of the National Institutes of Health (NIH) and National Science Foundation (NSF), and the Chairman, National Endowment for the Humanities (NEH), should address the underrepresentation of young, junior scholars as reviewers by targeted outreach efforts, at least experimentally.

    Agency Affected: National Science Foundation

  9. Status: Closed - Implemented

    Comments: NEH says it has increased the number of junior scholars in its database through a variety of means. NEH has found that young scholars "are often thorough and energetic reviewers and panelists."

    Recommendation: The Directors of the National Institutes of Health (NIH) and National Science Foundation (NSF), and the Chairman, National Endowment for the Humanities (NEH), should address the underrepresentation of young, junior scholars as reviewers by targeted outreach efforts, at least experimentally.

    Agency Affected: Department of Health and Human Services: Public Health Service: National Institutes of Health

  10. Status: Closed - Implemented

    Comments: NSF simplified and clarified its criteria. "The system will be improved only if the reviewers use the criteria when evaluating the proposal. Thus...criteria...must be formatted for maximal use." Guidance to reviewers now also asks reviewers to comment on the relative importance they gave to the two principal criteria. Guidance to staff asks that additional program-specific criteria be included in the announcement. Staff are also asked to mention the new criteria in any correspondence with the proposing communities. However, what is not clear is the extent to which NSF first identified any commonly used unwritten or informal decision rules before developing the new criteria.

    Recommendation: The Directors, NIH and NSF, and the Chairman, NEH, should identify any commonly used unwritten or informal decision rules that are applied by peer reviewers and, where feasible, formalize them or at least inform applicants of their importance.

    Agency Affected: National Science Foundation

  11. Status: Closed - Implemented

    Comments: PHS' 398 grant application kit is being revised and applicants will be instructed to include information on preliminary studies, and a tear-out request form for further information is being added. The Rating Grant Applications Committee is further reviewing this issue.

    Recommendation: The Directors, NIH and NSF, and the Chairman, NEH, should identify any commonly used unwritten or informal decision rules that are applied by peer reviewers and, where feasible, formalize them or at least inform applicants of their importance.

    Agency Affected: National Foundation on the Arts and Humanities: National Endowment for the Humanities

  12. Status: Closed - Implemented

    Comments: Beginning in FY 1995, NEH has requested and retained data on race/ethnicity and gender of applicants and reviewers. The task force recommended that the data be kept separately from all other records.

    Recommendation: The Chairman, NEH, should improve evaluation and oversight by retaining data on the race of applicants.

    Agency Affected: National Foundation on the Arts and Humanities: National Endowment for the Humanities

  13. Status: Closed - Not Implemented

    Comments: A task force examined mail review and recommended that the different programs continue to use the type of review "that suits their needs." There is no plan to increase the use of mail reviewers, but rather to simply improve the technology used to communicate those reviews.

    Recommendation: The Chairman, NEH, should improve the level of relevant expertise in reviews by making greater use of mail reviewers.

    Agency Affected: Department of Health and Human Services: Public Health Service: National Institutes of Health

  14. Status: Closed - Implemented

    Comments: Action has been completed.

    Recommendation: The Director, NSF, should increase efforts to calibrate proposal ratings among reviewers through information provided in advance and discussions of examples at the convening of panels.

    Agency Affected: National Science Foundation

  15. Status: Closed - Implemented

    Comments: NSF is planning to collect some limited data on the inclusion of women and minorities among their external reviewers, in some directorates but not all. However, it is a very limited effort. Unfortunately, as NSF officials GAO interviewed pointed out, the reason the representativeness of the internal or panel reviews had improved while that of the external reviewers had not, was because it was closely monitoring the composition of panels but not of the external, mail reviewers. This is particularly important to those fields relying heavily or exclusively on external review.

    Recommendation: The Director, NSF, should address reviewer representativeness by more closely monitoring the inclusion of women and minorities among external reviewers.

    Agency Affected: Department of Health and Human Services: Public Health Service: National Institutes of Health

  16. Status: Closed - Implemented

    Comments: The percentage of NSF reviews done solely by mail review has decreased, while the proportion of reviews done by both panels and mail review has increased.

    Recommendation: The Director, NSF, should address the potential for bias arising out of extensive personal relationships between reviewers and applicants by increasing the use of panels, where feasible.

    Agency Affected: National Foundation on the Arts and Humanities: National Endowment for the Humanities

  17. Status: Closed - Not Implemented

    Comments: NIH disagrees with the recommendation, arguing that the same results can be achieved through its program of monitoring for discrimination. However, by destroying the data, NIH makes full congressional oversight impossible.

    Recommendation: The Director, NIH, should improve evaluation and oversight by retaining data on scores given by individual panelists and the race and gender of individual applicants.

    Agency Affected: Department of Health and Human Services: Public Health Service: National Institutes of Health

  18. Status: Closed - Implemented

    Comments: NIH is making somewhat greater use of subpanels. It is allowing ad hoc reviewers to vote in some circumstances and has changed its guidance to SRAs to distribute mail reviews to more members of the panel.

    Recommendation: The Director, NIH, should produce a better match of panelists' area of expertise to that of the proposals by making greater use of subpanels and more fully integrating the work of ad hoc mail reviewers into the panel process.

    Agency Affected: National Foundation on the Arts and Humanities: National Endowment for the Humanities

  19. Status: Closed - Implemented

    Comments: Panelists will continue to be instructed to maintain their discussion within the scope of the criteria.

    Recommendation: The Directors, NIH and NSF, and the Chairman, NEH, should identify any commonly used unwritten or informal decision rules that are applied by peer reviewers and, where feasible, formalize them or at least inform applicants of their importance.

    Agency Affected: National Science Foundation

  20. Status: Closed - Implemented

    Comments: NIH established a database of potential reviewers that includes a higher proportion of junior faculty than are currently serving on review panels. Review staff are being actively encouraged to make increasing use of this database in selecting reviewers, regardless of age or academic rank.

    Recommendation: The Directors of the National Institutes of Health (NIH) and National Science Foundation (NSF), and the Chairman, National Endowment for the Humanities (NEH), should address the underrepresentation of young, junior scholars as reviewers by targeted outreach efforts, at least experimentally.

    Agency Affected: National Foundation on the Arts and Humanities: National Endowment for the Humanities

 

Explore the full database of GAO's Open Recommendations »

Oct 24, 2014

Oct 20, 2014

Oct 9, 2014

Oct 8, 2014

Oct 2, 2014

Sep 30, 2014

Sep 26, 2014

Sep 25, 2014

Sep 19, 2014

Looking for more? Browse all our products here