Program Evaluation:

OMB's PART Reviews Increased Agencies' Attention to Improving Evidence of Program Results

GAO-06-67: Published: Oct 28, 2005. Publicly Released: Oct 28, 2005.

Additional Materials:

Contact:

Nancy R. Kingsbury
(202) 512-2700
contact@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

The Office of Management and Budget (OMB) designed the Program Assessment Rating Tool (PART) as a diagnostic tool to draw on program performance and evaluation information for forming conclusions about program benefits and recommending adjustments to improve results. To assess progress in improving the evidence base for PART assessments, GAO was requested to examine (1) agencies' progress in responding to OMB's recommendations to evaluate programs, (2) factors facilitating or impeding agencies' progress, and (3) whether agencies' evaluations appear to be designed to yield the information on program results that OMB expects.

GAO examined agency progress on 20 of the 40 evaluations OMB recommended in its PART reviews at four federal agencies: the Department of Energy, Department of Health and Human Services, Department of Labor, and Small Business Administration. About half the programs GAO reviewed had completed an evaluation in the 2 years since those PART reviews were published; 4 more were in progress and 3 were still being planned. Program restructuring canceled plans for the remaining 2 evaluations. Several agencies struggled to identify appropriate outcome measures and credible data sources before they could evaluate program effectiveness. Evaluation typically competed with other program activities for funds, so managers may be reluctant to reallocate funds to evaluation. Some agency officials thought that evaluations should be targeted to areas of policy significance or uncertainty. However, all four agencies indicated that the visibility of an OMB recommendation brought agency management attention--and sometimes funds--to get the evaluations done. Moreover, by coordinating their evaluation activities, agencies met these challenges by leveraging their evaluation expertise and strategically prioritizing their evaluation resources to the studies that they considered most important. Because the OMB recommendations were fairly general, agencies had flexibility in interpreting the kind of information OMB expected. Some program managers disagreed with OMB on the purpose of their evaluations, their quality, and the usefulness of "independent" evaluations by third parties unfamiliar with their programs. Agency officials concerned about an increased focus on process said that they were more interested in learning how to improve program results than in meeting an OMB checklist. Since a few programs did not discuss their evaluation plans with OMB, it is not certain whether OMB will find their ongoing evaluations useful during the programs' next PART review. GAO concludes that the PART review process stimulated agencies to increase their evaluation capacity and available information on program results. Further, agencies are likely to design evaluations to meet their own needs--that is, in-depth analyses that inform program improvement. If OMB wants evaluations with a broader scope, such as information that helps determine a program's relevance or value, it will need to take steps to shape both evaluation design and execution. Finally, because agency evaluation resources tend to be limited, they are most usefully focused on important areas of uncertainty. Regular performance reporting is key to good management, but requiring all federal programs to conduct frequent evaluation studies is likely to result in superficial reviews of little utility and to overwhelm agency evaluation capacity.

Recommendations for Executive Action

  1. Status: Closed - Implemented

    Comments: OMB reports that agencies are now required to brief their appropriators and authorizers on past, completed, and planned PART assessments, including planned program evaluations. This represents partial implementation of our recommendation because it does not clearly distinguish OMB PART assessments from program evaluation studies, nor recognize the need to discuss evaluation plans and criteria with congressional stakeholders to ensure that their findings will be timely, relevant, credible, and used to inform decisions.

    Recommendation: OMB should encourage agencies to discuss their plans for program evaluations--especially those in response to an OMB recommendation--with OMB and with congressional and other program stakeholders to ensure that their findings will be timely, relevant, and credible and that they will be used to inform policy and management decisions.

    Agency Affected: Executive Office of the President: Office of Management and Budget

  2. Status: Closed - Not Implemented

    Comments: OMB reports that it has taken no formal actions in this area but that discussion of agency evaluation resources largely occur as part of agency budget discussions. OMB reports continuing discussions with agencies and outside stakeholders on the need to invest in rigorous evaluations, has offered agencies (in 2009 and 2010) additional funds for rigorous evaluations, but does not plan to address how agencies should allocate evaluation resources among its programs.

    Recommendation: OMB should engage in dialogue with agencies and congressional stakeholders on a risk-based allocation of scarce evaluation resources among programs, based on size, importance, or uncertain effectiveness, and on the timing of such evaluations.

    Agency Affected: Executive Office of the President: Office of Management and Budget

  3. Status: Closed - Implemented

    Comments: OMB sponsored workshops for OMB and agency staff that clarified that experimental research designs involving randomization were not the only acceptable method for evaluating federal program effectiveness and might not be feasible in some cases. OMB also sponsored a workshop where agency evaluators discussed the applicability of alternative evaluation designs.

    Recommendation: OMB should continue to improve its PART guidance and training of examiners on evaluation to acknowledge a wide range of appropriate methods.

    Agency Affected: Executive Office of the President: Office of Management and Budget

 

Explore the full database of GAO's Open Recommendations »

Dec 22, 2014

Dec 18, 2014

Dec 17, 2014

  • government icon, source: Eyewire

    State and Local Governments' Fiscal Outlook:

    2014 Update
    GAO-15-224SP: Published: Dec 17, 2014. Publicly Released: Dec 17, 2014.

Dec 3, 2014

Nov 14, 2014

Nov 13, 2014

Nov 12, 2014

Oct 31, 2014

Oct 30, 2014

Oct 27, 2014

Looking for more? Browse all our products here