The Logic of Evaluating Program Effects

In looking for the impact of Government programs, GAO evaluators must usually sort through a variety of competing explanations for observed phenomena. There is a logic to the sorting process which the evaluators can use to guide their reasoning. Two important features of that logic were reviewed by looking at the problems of evaluating the effect of a crime control program. The evaluator must: (1) compare what is observed to happen when a Government program is in operation with an estimate of what would have happened in the absence of the program; and (2) identify possible causes of observed effects and distinguish those caused by the program. For the crime control program, the evaluator determines the effect by contrasting the crime rate during program operation with an estimate of what the rate would have been without the program. There are many ways to arrive at the estimate, but these methods vary in their reliability. Random experiments are usually considered to be the best approach. Unfortunately, random experiments are not easily applied to many broad Government programs. In measuring the effects of a crime control program, all possible causes for a change in the crime rate must be isolated. There will often be a number of plausible explanations, and the evaluator must assume that each, to some extent, accounts for the observed effects. The important point is that ways to rule out alternative causes must be planned in advance of doing an evaluation. There is no single approach to evaluating the effects of Government programs. However, the credibility of the evaluation depends upon adhering to the logic of cause and effect.