Experienced Agencies Follow a Similar Model for Prioritizing Research
GAO-11-176, Jan 14, 2011
Amid efforts to improve performance and constrain spending, federal agencies are being asked to expand the use of rigorous program evaluation in decision-making. In addition to performance data, indepth program evaluation studies are often needed for assessing program impact or designing improvements. Agencies can also use their evaluation resources to provide information needed for effective management and legislative oversight. GAO was asked to study federal agencies with mature evaluation capacity to examine (1) the criteria, policies, and procedures they use to determine programs to review, and (2) the influences on their choices. GAO reviewed agency materials and interviewed officials on evaluation planning in four agencies in three departments with extensive evaluation experience: Education, Health and Human Services (HHS), and Housing and Urban Development (HUD). HHS and HUD agreed with the description of how they plan evaluations. HHS noted that the optimal location of evaluation units will vary with the circumstances and purpose of evaluations. HUD felt the draft report did not emphasize enough the influence of the appropriations process. GAO has added text to note its influence on evaluation planning. Education provided technical comments.
Although no agency GAO reviewed had a formal policy describing evaluation planning, all followed a generally similar model for developing and selecting evaluation proposals. Agencies usually planned an evaluation agenda over several months in the context of preparing spending plans for the coming fiscal year. Evaluation staff typically began by consulting with a variety of stakeholders to identify policy priorities and program concerns. Then with program office staff they identified the key questions and concerns and developed initial proposals. Generally, the agencies reviewed and selected proposals in two steps: develop ideas to obtain initial feedback from senior officials and develop full-scale evaluation proposals for review and approval. The four general criteria these mature agencies use to plan evaluations were remarkably similar: (1) strategic priorities representing major program or policy area concerns or new initiatives, (2) program-level problems or opportunities, (3) critical unanswered questions or evidence gaps, and (4) the feasibility of conducting a valid study. The agencies' procedures differed on some points. External parties' participation in evaluation planning may reflect these agencies' common reliance on nonfederal program partners. Only the offices GAO reviewed in HHS' Centers for Disease Control and Prevention held formal competitions to rank-order proposals before submitting them for approval; in other agencies, senior officials assessed proposals in a series of discussions. When evaluation authority and funds are tied to the program, evaluators generally choose not which programs to evaluate but which research questions to answer. Sometimes this resulted in a program's never being evaluated. Evaluation units at higher organizational levels conducted a wider range of analytic activities, consulted more formally with program offices, and had less control over approvals. The Congress influences an agency's program evaluation choices through legislating evaluation authority, mandating studies, making appropriations, and conducting oversight. GAO concludes that (1) all four criteria appear key to setting an effective evaluation agenda that provides credible, timely answers to important questions; (2) most agencies could probably apply the general model in which professional evaluators iteratively identify key questions in consultation with stakeholders and then scrutinize and vet research proposals; (3) agencies could adapt the model and decide where to locate evaluation units to meet their own organizational and financial circumstances and authorities; and (4) agencies' reaching out to key program and congressional stakeholders in advance of developing proposals could help ensure that their evaluations will be used effectively in management and legislative oversight. GAO makes no recommendations.