This is the accessible text file for GAO report number GAO-03-867R 
entitled 'Technical Assessment of Zhao and Thurman's 2001 Evaluation of 
the Effects of COPS Grants on Crime' which was released on July 14, 
2003.

This text file was formatted by the U.S. General Accounting Office 
(GAO) to be accessible to users with visual impairments, as part of a 
longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov.

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately.

United States General Accounting Office:

Washington, DC 20548:

June 12, 2003:

The Honorable F. James Sensenbrenner 
Chairman 
Committee on the Judiciary 
House of Representatives:

Subject: Technical Assessment of Zhao and Thurman's 2001 Evaluation of 
the Effects of COPS Grants on Crime:

Dear Mr. Chairman:

Community Oriented Policing Services (COPS) is a federal public safety 
program whose goals are to add officer positions to the streets of 
communities nationwide and to promote community policing. Since the 
program's inception in 1994, local law enforcement agencies have 
received billions of dollars in grants to hire additional officers, 
acquire technology and civilian personnel, and implement innovative 
crime-prevention programs. To receive COPS grants, agencies are 
expected to implement or enhance community policing strategies 
illustrating community partnerships, problem solving, and 
organizational commitment. Given the large expenditures of funds, it is 
important for policy makers, among others, to have sound information on 
the effectiveness of the COPS program in reducing crime. You asked us 
to review one evaluation of the effectiveness of the COPS program--by 
Zhao and Thurman[Footnote 1]--and to render an assessment of its 
quality. In this report, we provide information on the extent to which 
this particular study's conclusions are supported by the data the 
researchers used and the analyses they conducted. GAO statisticians and 
methodology specialists reviewed the study using standard and widely 
accepted statistical and social science research principles.

Our assessment of Zhao and Thurman's work cannot be construed to be an 
assessment of the COPS program itself. Since we have not reviewed the 
quality of any other COPS evaluation or conducted an independent 
evaluation of the program, we have no basis to judge whether or not the 
program has been effective in achieving its stated goals. It is also 
important to note that these types of aggregate level analyses that are 
intended to assess program effectiveness are extremely difficult to 
execute successfully, in part, because direct measures of important 
variables are not always available.

We conducted our review of Zhao and Thurman's study during a 3-week 
period in May 2003. In addition to reviewing Zhao and Thurman's 
December 2001 report, we reviewed a November 2002 journal article by 
Zhao, Scheider, and Thurman based on the same study,[Footnote 2] 
reviewed a May 2003 draft of an updated COPS study by the same authors, 
and discussed data and statistical issues with these researchers in a 
telephone call on May 27. In this report, we focus the majority of our 
comments on Zhao et al.'s earlier COPS study (reported in December 2001 
and November 2002). We discuss differences between the earlier study 
and the May 2003 follow-up study in a section at the end of this 
report. For ease of presentation, we refer to their original work as 
the "2001 study.":

Background:

The Public Safety Partnership and Community Policing Act of 
1994[Footnote 3] authorized $8.8 billion in grants to be awarded to law 
enforcement agencies for fiscal years 1995 to 2000. Focused on crime-
prevention, the act required, among other things, that half the grants 
go to law enforcement agencies serving populations of 150,000 or less. 
The act also required that grantees not supplant state and local 
funding, but rather use the federal funds for additional law 
enforcement beyond what would have been available without a grant. The 
Attorney General created the Office of Community Oriented Policing 
Services to administer the grant programs and advance community 
policing across the country.

The COPS office is tasked with promoting community policing through a 
variety of types of grants, including:

* Hiring grants, which are used to fund the hiring of additional police 
officers. Through its Universal Hiring Program, the COPS program 
provides funding directly to local, state, and tribal jurisdictions. 
The funding provides up to 75 percent of the salaries and benefits for 
new officers for 3 years up to a maximum of $75,000 per officer. 
According to the COPS Office, 71,192 officers were funded and 63,592 
officers were hired through hiring grants as of July 26, 2002. The COPS 
Office estimated that hiring grant awards totaled about $5.6 billion as 
of June 3, 2003.

* Making Officer Redeployment Effective (MORE) grants, which are used 
to fund up to 75 percent of the total cost of acquiring new 
technologies and equipment and the hiring of civilians for 1 year. 
These are intended to allow police to spend more time patrolling the 
streets instead of on administrative and support tasks. According to 
the COPS Office, 24,436 full-time equivalent staff were redeployed 
through MORE grants as of July 26, 2002. The COPS Office estimated that 
MORE grant awards totaled about $1.3 billion as of June 3, 2003.

* Innovative grants, which are used to promote innovative approaches to 
solving crime in specific areas such as domestic violence and drug 
abuse. The COPS Office estimated that innovative grant awards totaled 
$820 million as of June 3, 2003.

Results in Brief:

Our review of the 2001 study on the effects of COPS grants on crime 
rates indicated that the results of their study should be viewed as 
inconclusive. We believe that the study's limitations in data and 
methods are significant and preclude meaningful interpretation of the 
results. We cannot agree with Zhao et al. that their 2001 study shows 
that some COPS grants (hiring and innovative) significantly reduced 
crime because, among other things, important variables were omitted 
from their analyses, the analytic models were misspecified, and the 
sample of cities included in the study was limited. Further, we have 
concerns about the use of outdated census data for control variables. 
Aside from concerns about data and methods, we question whether the 
statistically significant crime reductions that Zhao et al. found are 
significant in a practical sense.

While we cannot agree with the Zhao et al.'s conclusions, we also 
cannot say that COPS grants are ineffective in reducing crime. A 
program's effects and researchers' ability to design studies that will 
accurately measure those effects are two different things. Other 
studies, which we have not reviewed, may have taken a more rigorous 
approach to assessing the effects of COPS grants on crime. We believe 
that a more rigorous study would incorporate, among other things, more 
reliable, valid, and complete measures; a more complete and 
generalizable sample of cities; and well-specified analytic models.

In written comments on a draft of this report, the Department of 
Justice's COPS Office and Zhao and Thurman generally disagreed with our 
findings. The comments reflected the view that our standards for 
critiquing Zhao et al.'s work were too stringent, that we were 
incorrect in concluding that their statistical models were 
misspecified, and that the statistical controls incorporated into their 
analytic models were sufficient to account for the types of missing 
data we identified as limitations of the study. In our response, we 
address why we continue to believe that these limitations render the 
findings of this particular study inconclusive.

Summary of Analysis and Results of the 2001 Study:

The 2001 study presented a statistical analysis of the effects of three 
types of grants--hiring, MORE, and innovation--on the reported rates of 
violent and property crimes over a 5-year period across 6,100 U.S. 
cities that received COPS grants. The analysis, which looked separately 
at cities with populations greater than 10,000 and those with 
populations less than 10,000, sought to determine how the reported 
crime rates varied as a function of the amount of COPS funds received.

The variables used in the 2001 study are presented in table 1, along 
with the averages and standard deviations for these variables across 
all cities included in the analysis.

Table 1: Averages Across All Cities from 1994 to 1999:

Dependent variables (1995-1999):

Violent crime rate (per 100,000 population); Average: 769.63; Standard 
deviation: 674.50.

Property crime rate (per 100,000 population); Average: 5,016.39; 
Standard deviation: 2,820.74.

Independent variables (1994-1998):

Hiring grants (per resident); Average: $2.38; Standard deviation: 3.72.

Innovative grants (per resident); Average: $0.42; Standard deviation: 
2.45.

MORE grants (per resident); Average: $0.65; Standard deviation: 1.45.

Demographic control variables:

% unemployment (1994-1998); Average: 4.97; Standard deviation: 2.17.

% minority (1990 census); Average: 30.40; Standard deviation: 23.32.

% single parent households (1990 census); Average: 10.59; Standard 
deviation: 4.09.

% young people ages 15-24 (1990 census); Average: 15.43; Standard 
deviation: 4.59.

% home owners (1990 census); Average: 56.92; Standard deviation: 14.62.

% people in same household for 5 or more years (1990 census); Average: 
50.66; Standard deviation: 10.03.

Source: Zhao et al., December 2001 and November 2002.

Note: Zhao et al. used weighted averages to estimate the means of COPS 
grants and control variables.

[End of table]

Zhao et al. found that hiring grants significantly reduced reported 
violent and property crimes in larger cities, but significantly 
increased those rates in smaller cities. They speculated that the 
addition of police officers in smaller cities could produce an increase 
in reported crime because, among other things, the increased 
interaction between police and the community can help residents feel 
more comfortable and willing to report crimes. Innovative grants also 
significantly reduced the reported violent and property crime rates in 
larger cities, but had no significant effect in smaller cities. MORE 
grants had no discernable effect in larger cities, or on reported 
violent crimes in smaller cities, but they significantly increased the 
rates of reported property crimes in the smaller cities. Zhao et al. 
concluded that innovative programs, which are targeted at specific 
crime problems or jurisdictions, had the strongest effect on reducing 
reported crime rates. They also observed that "crime reduction in the 
United States is not a unitary phenomenon" in light of the different 
effects found in large versus smaller cities.

Our Review Indicated Several Problems with the 2001 Study:

Our review revealed several problems with the 2001 study that cast 
doubt on the validity of the conclusions about the effectiveness of 
COPS grants. The problems we identified pertain to Zhao et al.'s 
interpretation of their findings, omission of important variables from 
the analysis, misspecifications in the analytic models used, and sample 
selection issues. We also had some concerns about the outdated nature 
of census data used as control variables in the 2001 study.

The Meaning of the Study's Findings Can Be Interpreted Differently:

The finding that COPS grants exerted different effects on crime 
patterns in large versus small cities led the researchers to observe 
that crime reduction is not a unitary phenomenon. While this may be the 
case, one can also conclude that the study's findings are equivocal, 
inconsistent, and inconclusive.

Further, while the crime-reducing effects that Zhao et al. found for 
hiring and innovative grants may have been statistically significant, 
they could also be characterized as quite small in a practical 
sense.[Footnote 4] Table 2 demonstrates this point by presenting a 
summary of Zhao et al.'s estimates.

Table 2: Estimates of the Effects of Three Types of COPS Grants, from 
Zhao et al. (2001):

[See PDF for table]

Source: GAO summary of Zhao et al. 2001 data.

Note: An asterisk (*) denotes that the estimated effect was 
statistically significant.

[End of table]

The coefficients in table 2 indicate how much each grant dollar spent 
per person in each city affected the rates of reported violent and 
property crimes; in other words, how much of a change in the reported 
violent and property crime rates we might expect if funding were 
increased by one dollar per resident. As shown in table 1, the average 
annual COPS innovative grant across all cities amounted to $0.42 per 
person, and the average rates of reported violent and property crimes, 
respectively, were about 770 and 5,016 per 100,000. These coefficients 
imply that if COPS funding in larger cities for innovative grants were 
doubled (from $0.42 to $0.84 per person), we would expect the violent 
crime rate to go down by 0.7 of 1 percent (from 770 to 765 per 
100,000).[Footnote 5] We would expect the reported property crime rate 
to go down by 0.4 of 1 percent (from 5,016 to 4,997 per 
100,000).[Footnote 6] As small as the effects are, there are reasons to 
question whether they accurately represent the expected returns on such 
an investment, and these reasons are listed below in general order of 
importance.

Important Variables Were Omitted from the Analysis:

While dummy variables were used in the 2001 study to control for 
unmeasured differences across counties, the only city-level variables 
in the analysis that were measured and explicitly controlled in the 
models of estimated COPS grant effects were (1) the 1994 crime rate and 
(2) the six demographic variables shown in table 1. Most conspicuously 
absent from these models is a measure of expenditures on police that 
were not derived from COPS grants. The researchers told us they did not 
include non-COPS funded police expenditures because such data are not 
available. Because the COPS program supports only a portion of police 
agency budgets, however, we believe the absence of any control for 
state and local expenditure to be a serious weakness.

Police departments that received COPS grants may have also received 
grants from other programs (such as Byrne grants). These amounts could 
be correlated with COPS funding amounts. For example, if a department 
is proficient in getting COPS funding, it may be proficient in getting 
other funding, as well. Without separating COPS funding from other 
types of funding that police agencies receive, we cannot be sure how 
much of an effect COPS grants by themselves have on crime reduction.

The study also lacked any measure of city size beyond the dichotomy 
(i.e., population smaller or larger than 10,000) used to split the 
sample of cities prior to model estimation. Other omitted measures 
include such socioeconomic variables as per capita income and percent 
male. County dummy variables controlled for some of the problems 
associated with omitted variables, but they would not control 
effectively for variables that differed across cities within counties, 
or variables that changed within counties over time. For example, if 
state and local expenditures on police varied across cities in a given 
county, using dummy variables to represent counties would not take 
these differences or changes into account in estimating the independent 
effect of COPS grants.

Misspecifications in the Analytic Models:

The models employed in Zhao et al.'s analyses are two-factor fixed 
effects models that employ 2,674 dummy variables representing the 
counties and 5 dummy variables representing the years included in the 
analysis. These dummy variables controlled for unmeasured variability 
across counties and over time, and they supplemented the controls for 
prior rate of crime and the 6 demographic variables described above. 
These models and the estimation procedures they involve are fairly 
sophisticated, but since the data on crime rates and COPS funds were 
measured at the city level, we believe that unmeasured variability 
would have been more effectively controlled had dummy variables been 
used to distinguish cities, instead of the counties in which the cities 
were located.[Footnote 7] With dummy variables representing counties, 
any unmeasured and systematic variability across cities within the same 
county remained uncontrolled and a potential source of bias in the 
parameters representing the effects of the COPS grants estimated in the 
models.

Sample Selection Limited:

Zhao et al.'s analysis is focused only on COPS program grants used to 
fund local city police departments. Their report indicates that other 
law enforcement agencies, such as state and county police agencies; 
sheriffs' offices; campus police; and special purpose law enforcement 
agencies such as court, forest, and park police, among others, were 
excluded from their study. Since these other agencies accounted for 
4,891 (or 40 percent) of the 12,070 law enforcement agencies receiving 
COPS grant awards from 1994 to 1998, Zhao et al.'s study omitted a 
large portion of COPS grant recipients. Further, there is likely to be 
considerable overlap across jurisdictions receiving COPS grants (cities 
within counties, campus police within city jurisdictions).[Footnote 8]

According to Zhao et al., the sample of cities included in their study 
represented a subset of 6,100 of the 7,179 cities whose local city 
police departments received COPS grants at some point during the period 
from 1994 to 1998.[Footnote 9] The researchers deleted 535 cities with 
populations less than 1,000, and 544 cities that lacked Uniform Crime 
Reports (UCR) data.[Footnote 10],[Footnote 11] Four states (Delaware, 
Illinois, Kansas, and Montana) contributed only 8 cities between them 
owing to missing UCR data. These omissions may have affected the 
study's results. Of greater concern, however, is the omission of the 
potentially large number of cities that received no COPS funding at 
all. We believe that cities with no COPS funding should have been 
included in the analyses in order avoid sample selection problems and 
ensure that the results were generalizable across all cities.[Footnote 
12]

Concerns about Measures of Demographic Variables.

While the rates of violent crimes and property crimes were measured and 
allowed to vary in each of the 5 years from 1994 to 1998, in the 2001 
study at least 5 of the 6 demographic variables were derived from the 
1990 census and fixed at their 1990 levels. We believe the 1990 figures 
would be a poor basis for estimates because in many cities, the 
demographic characteristics of residents in 1990 would be expected to 
be quite different from those in the mid-to late-1990s; and in all 
cities, these time-invariant estimates would fail to account for the 
significant demographic changes that may have occurred over time. It 
was not entirely clear to us how the unemployment data derived from the 
U.S. Department of Labor Statistics for the years 1994 to 1998 were 
used in these models. However, it too represented a potentially poor 
measure of unemployment in many cities. This is because data are not 
available for cities with populations less than 25,000, and county-
level rates were used for those cities instead.

Comments on Zhao et al.'s Draft Updated COPS Study:

Our previous comments pertain to the unpublished 2001 study by Zhao and 
Thurman and the 2002 publication by Zhao, Scheider, and Thurman which 
resulted from the study and which was virtually identical to the 
unpublished study in terms of the primary results that were reported. 
That study, as we noted previously, relied on data from 6,100 cities 
for which COPS grant data for the years 1994 to 1998, and UCR crime 
data for the years 1994 to 1999, could be obtained. After reviewing 
that work, we received a draft updated report from those authors that 
re-estimated the effects of COPS grants on crime rates using data from 
an additional year (e.g., COPS grant data for 1994-1999 and UCR data 
for 1994-2000)[Footnote 13] and models that incorporated updated 2000 
census data and allowed the demographic characteristics to vary over 
time. While these newer estimates, like those in the 2001 and 2002 
reports, were derived from models that used county dummy variables, we 
also received from the researchers additional information that showed 
how results compared when they used dummy variables representing cities 
in place of the county dummy variables.

These updated results are shown in table 3, along with the results from 
the researchers' prior study. The researchers have asserted, both in 
the draft updated report and in their conversations with us, that these 
updated results are largely consistent with the previously published 
results, and in a general sense we agree with this. That is, with or 
without the newer data, regardless whether demographic factors are 
allowed to vary, and regardless whether county or city or dummy 
variables are used, both studies found (1) no evidence that COPS grants 
have diminished the crime rates in cities with populations less than 
10,000, and (2) some evidence that they have done so in larger cities. 
Apart from this general observation, however, the results of the two 
studies are inconsistent in that the size and significance of some of 
the estimated effects of COPS grants differed under alternative 
specifications. For example, when updated data and the time varying 
covariates were used, the estimated effects of innovative grants on 
violent and property crimes in large cities declined in size to less 
than half of the prior estimates, while the effects of MORE grants 
increased more than 10-fold, and became statistically significant in 
the case of property crimes.[Footnote 14]

Since these newer results have not been finalized, it is premature for 
us to make a final determination of their validity and usefulness. The 
researchers are to be commended for the considerable effort they made 
to determine how reliable and robust the estimated effects of the 
different COPS grants were over time, and under alternative 
specifications. Nonetheless, the newer study that we reviewed had some 
of the same limitations as the 2001 study. Specifically, the newer 
study (1) omitted important variables, including measures of 
expenditures on police apart from COPS grants, (2) omitted a large 
number of cities that did not receive COPS grants, and (3) did not 
control for the effect of city size on crime in a more refined fashion 
than dichotomizing city populations. Our review of the results of the 
newer analyses has not fundamentally altered our view that the 
estimated effects of COPS grants on reported violent and property 
crimes were small in a practical sense. Again, it is important to note 
that this does not imply that COPS grants do not have positive effects 
in reducing crime; only that it is hard to reach firm conclusions about 
their effects from the particular studies we reviewed. Our technical 
assessment of Zhao et al.'s work is not a commentary on the 
effectiveness of the COPS program.

Table 3: Zhao et al.'s Estimates of the Effects of Three Types of COPS 
Grants with Dummy Variables Representing Counties in (a) 2001 Study 
Using 1994-1999 Data and (b) Draft Updated Study Using 1995-2000 Data:

[See PDF for image]

Source: GAO summary of Zhao et al.'s 2001 and updated studies.

Note: An asterisk (*) denotes that the estimated effect was 
statistically significant.

[End of table]

Agency Comments and Our Evaluation:

The Acting Deputy Director of the COPS Office and Professors Zhao and 
Thurman provided us with written comments on a draft of this report. 
Their comments contained a number of points that disagreed with the 
limitations we identified in our assessment. The comments reflected the 
view that we (1) applied an overly stringent standard to the study's 
design and failed to consider the fact that this study was better and 
more comprehensive than previous research on the subject; (2) were 
incorrect in concluding that their statistical models were misspecified 
and did not control for the effect of missing police expenditure data; 
(3) were ill-advised in stating that including data on cities' access 
to grants other than COPS grants would have improved the estimates of 
COPS grant effects; (4) were ill-advised in stating that including data 
on such socioeconomic variables as percentage of the population that is 
male would have improved the estimates of COPS grant effects; (5) were 
incorrect in stating that including data on COPS-funded jurisdictions 
within cities, such as university police, would have improved the 
estimates of COPS grant effects; and (6) were ill-advised in stating 
that including police departments in cities that did not receive COPS 
funding would have improved the estimates of COPS grant effects. We 
continue to disagree with the researchers on these key points and 
discuss our reasons below.

First, with respect to the assertion that our standards were too high 
and that we did not consider the advances made by this study, we would 
reiterate that the purpose of our assessment was to determine the 
extent to which the conclusions of this particular COPS study were 
supported by the data used and analyses conducted. Because we were 
asked to review this single study and did not have time to review any 
others, we cannot comment on whether and how this study's approach to 
evaluating the effectiveness of the COPS program may have been an 
incremental improvement over other similar efforts. We acknowledge in 
the introduction to this report that it is extremely difficult to 
assess program effectiveness via aggregate level analyses. We also 
believe that the researchers should be commended for their efforts, 
which involved merging data on more than 6,000 towns and cities over a 
multi-year period from four different sources and using sophisticated 
methods to analyze those data under a variety of specifications. But, 
in our estimation, the problems that we identified with the research 
make the results more suggestive than conclusive.

Second, with respect to the assertion that the statistical models were 
both correctly specified and sufficiently controlled for the effect of 
missing data on police expenditures, we do not believe this was the 
case. Zhao et al. believe that we are unjustifiably critical of their 
having used county rather than city dummy variables in estimating the 
effects of COPS grants on crime rates. They point out that they ran 
both their 2001 and 2003 analyses using both city and county dummy 
variables, and the results of the two types of analyses did not differ 
substantially.[Footnote 15] While the models incorporating county or 
city dummy variables do, as the authors assert, explain a sizable 
portion (between 64 percent and 86 percent) of the variation in 
reported crime rates across cities over time, this is not surprising 
and is largely attributable to the very large number of dummy variables 
included in their models. The proportion of variance explained, 
however, does not necessarily imply that the estimates of the effects 
of COPS grants were unbiased. The authors, in our opinion, are mistaken 
in their claim that the use of dummy variables controls for the effects 
of all unmeasured differences between cities and over time. That is, 
the county dummy variables do not control for unmeasured differences 
between cities within counties, and even the combination of city and 
year dummy variables do not control for differences within cities over 
time, unless the changes in all cities are similar. Crime rates in 
cities did not show similar changes over time,[Footnote 16] however, 
and there are many factors that can change in cities from one year to 
the next in ways that might affect crime rates. For example, 
fluctuations in local, state, and other expenditures on police could 
produce changes in crime rates within cities over time, and the failure 
to control for such factors can seriously bias the estimates of the 
effects of COPS grants.

Third, with respect to the assertion that omitting data on cities' 
access to grants other than COPS grants probably did not affect the 
results, we are not convinced. We agree that data on grants that cities 
receive are not readily available. However, we believe that information 
on at least major grant programs could be obtained from the Office of 
Justice Programs. To the extent that cities that receive COPS grants 
may be more likely to receive other types of grants, omitting 
consideration of other grants that are also targeted at reducing crime 
may lead to an overestimation of the effects of COPS grants. By 
restricting their attention to COPS grants awarded to city and local 
police, the researchers investigated the effects of only a portion of 
all COPS grants. They ignored the effects of other grants and of state 
and local expenditures, generally, and therefore increased the 
potential for obtaining biased estimates of COPS grant effects.

Fourth, with respect to the assertion that the study's results were not 
impaired by the omission of such socioeconomic variables as percentage 
of the city population that is male, we disagree with the researchers 
that this is not problematic. They assert that (1) the dummy variables 
in their statistical models controlled for the effects of socioeconomic 
variables other than those in their analyses, (2) a city's male 
population should not significantly affect the estimated effects of 
COPS grants on crime, (3) the socioeconomic variables included in the 
analyses were sufficient and grounded in widely accepted social 
disorganization theory, and (4) problems of multicollinearity[Footnote 
17] could have arisen had they included additional socioeconomic 
variables. As with police expenditures, we maintain that data on 
factors affecting crime rates that vary across cities and over time 
should be included in analyses, and may not be sufficiently controlled 
by statistical models that use dummy variables to control for 
unmeasured differences. While we do not know whether and how COPS grant 
amounts to cities may be associated with the socioeconomic 
characteristics of city residents, the literature indicating a gender 
difference in crime is extensive.[Footnote 18] To the extent that 
socioeconomic characteristics affect crime rates, and to the extent 
that cities that received COPS grants may have different socioeconomic 
characteristics, we believe it would be wise to incorporate such 
variables into models to lessen any potential bias in the estimates of 
the COPS grants on crime. Since this study was intended to be an 
evaluation of the effects of COPS grants on crime and not as a test of 
social disorganization theory, we do not believe that limiting the 
socioeconomic control variables to those dictated by this particular 
theory was warranted. Finally, with 36,000 observations in their study, 
we do not believe that multicollinearity would have been a problem had 
additional socioeconomic variables been included in the analyses.

Fifth, with respect to the assertion that including data on COPS-funded 
agencies within cities would not have improved the estimates of COPS 
grant effects, we continue to believe that this cannot be known. Zhao 
and Thurman state that there is no meaningful way to include such 
agencies--for example, park and university police--in their statistical 
models because the jurisdictions overlap. They note that it was neither 
necessary nor possible to estimate the effects of such agencies on 
crime rates because they report crime incident data to the Federal 
Bureau of Investigation separately and because census data for them are 
not readily available. We maintain that by restricting their attention 
to crimes reported to local and city police departments, the 
researchers are investigating the effect of only a portion of all COPS 
grants and are looking at only a subset of all crimes reported. Again, 
we do not know whether these restrictions result in an overestimate or 
underestimate of the effect of COPS grants on crimes, but they can 
potentially bias their estimates. We acknowledge that data may not be 
readily available for such an analysis, but that does not mean they 
cannot be collected or that they are unimportant.

Sixth, with respect to the assertion that including data on police 
departments in non-COPS funded cities would not have improved the 
estimates of COPS grant effects, we continue to disagree. Zhao and 
Thurman note that because small cities are more likely than large 
cities to not receive COPS funding, including nonfunded agencies in 
their analysis could bias the findings towards showing an effect of 
COPS grants. It is our view that missing cases, except when they are 
missing at random, should be regarded as problematic. The 6,100 
agencies that Zhao et al. analyzed represented about 85 percent of the 
COPS-funded city and local police departments, 51 percent of the total 
number of COPS-funded agencies, and 36 percent of the agencies that 
participate in the UCR system. Some of these exclusions may have been 
unavoidable, but their cumulative impact is likely to be non-
negligible.

We do not know how or to what extent the findings that Zhao et al. 
obtained would change if the limitations that we identified in our 
assessment were successfully resolved. We do know, however, that while 
Zhao et al. may have performed the most sophisticated and advanced 
research on the topic, drawing inferences or making policy decisions 
about COPS grant effects from this work are unwarranted at this time. 
Indeed, Zhao and Thurman are themselves continuing this work, an 
indication that they also believe refinements are needed.

The comments from the COPS office and the researchers are reproduced in 
the enclosure to this report. The COPS Office also provided us with 
technical comments, which we incorporated in the report as appropriate.

------------:

As agreed with your office, unless you publicly announce the contents 
of this report earlier, we plan no further distribution of it until 30 
days from the date of this report. We will then send copies of the 
report to the Attorney General and will make copies available to others 
upon request. In addition, the report will be available at no charge on 
GAO's web site at http://www.gao.gov.

If you have any questions about this report, please contact me at (202) 
512-8777. The key contributors to this report were David Alexander, 
Carl Barden, Evi Rezmovic, and Douglas Sloane.

Sincerely yours,

Laurie E. Ekstrand 
Director, 
Homeland Security and Justice Issues:

Nancy Kingsbury 
Managing Director, 
Applied Research and Methods Issues:

Signed by Laurie E. Ekstrand and Nancy Kingsbury: 

Enclosures - 2:

[End of section]

Enclosure I:

U.S. Department of Justice:

Office of Community Oriented Policing Services (COPS) June 5, 2003:

VIA FACSIMILE and ELECTRONIC MAIL:

Laurie E. Ekstrand:

Director, Homeland Security and Justice 
United States General Accounting Office 
Washington, DC 20548:

Nancy Kingsbury:

Managing Director, Applied Research and Methods 
United States General Accounting Office 
Washington, DC 20548:

Dear Ms. Ekstrand and Ms. Kingsbury:

The U.S. Department of Justice Office of Community Oriented Policing 
Services (COPS Office) thanks the GAO for conducting an audit of the 
methodological approach used in the December, 2001 study, "The National 
Evaluation of the Effect of COPS Grants on Crime from 1994 to 1999." We 
welcome the GAO recommendations and appreciate the opportunity to 
respond to the review.

COPS Office Grant Programs:

Since its creation, the COPS Office has assisted nearly 13,000 of the 
nation's approximately 18,000 law enforcement agencies in implementing 
community policing. The COPS Office has invested $9.6 billion to add 
officers to the nation's streets and schools, enhance crime-fighting 
technology, support crime prevention initiatives, and provide training 
and technical assistance. Specifically, to date, the COPS Office has 
provided funding to hire or redeploy 116,000 police officers and 
sheriff's deputies. Currently, over 83,000 of them are on the beat. In 
addition, the COPS Office has funded 6,000 school resource officers 
(SROs), provided $1.1 billion in crime-fighting technology and $200 
million in law enforcement assistance to Indian Country. Moreover, 
through a national network of COPS Regional Community Policing 
Institutes (RCPIs), the COPS Office has provided training for 209,000 
law enforcement personnel, government leaders, and community members in 
various community policing strategies. This substantial support for 
State, local, and tribal law enforcement has produced real results for 
communities across the nation.

National Evaluation of the Effect of COPS Grants:

The COPS Office is committed to the continuous evaluation of its 
programs and their impact. In 2000, the COPS Office funded a study 
specifically to assess the impact of COPS funding on crime and how, 
within the bounds of legislation, COPS can develop grant programs that 
best support State, local and tribal law enforcement. The study was 
also designed to respond to the requirements of the Government 
Performance and Results Act (GPRA), requiring Federal agencies to 
collect and analyze data on the impact of their programs and 
activities.

As the GAO report notes, the social science involved in assessing 
program effectiveness is difficult, in part because direct measures of 
important variables are not always available. With this in mind, the 
COPS Office engaged the University of Nebraska (Omaha) and two 
distinguished researchers to conduct the study. After almost 12 months 
of research and analyses, including rigorous, independent peer reviews 
by well-respected social science experts, the researchers concluded 
that COPS hiring and innovative grant programs have a significant crime 
reducing effect on the vast majority of the population of the United 
States.

Extensive Peer Review Process by Social Science Experts:

The study was subject to a rigorous review of its methodology and 
results. A total of eleven independent social scientists and 
statisticians peer reviewed the study. Specifically, the initial study 
was submitted to three peer reviewers. Thereafter, five additional 
social scientists peer reviewed an article based on the study that was 
published in Criminology and Public Policy, a prestigious criminal 
justice journal. Finally, the researchers' pending update of the study, 
referenced by the GAO, has likewise been submitted to three experts for 
peer review. Eight of these eleven experts determined that the 
methodological approach taken by the researchers was sound. One 
reviewer did not reach a final conclusion, and the two remaining 
individuals offered recommendations, many of which are identical to 
those now raised by the GAO.

In response to the external peer review, the researchers either amended 
the study or determined that the recommendations would have an 
insignificant or, in one case, a detrimental effect on the validity of 
the study. The researchers' extensive reasons as to why the 
recommendations were not included, why the recommendations would not 
likely result in an improved analytical model or why their inclusion 
may unfairly skew the results in favor of finding an impact of COPS 
grants on crime, are set forth in the attached response from the 
researchers to the GAO report. For example, the researchers point out 
that updated 2000 Census data was not available at the time of the 
study. In addition, the absence of police expenditure data was 
addressed by including the 1994 crime rate (as a reflection of the 
ability to control crime with local resources) and county and city 
dummy variables (standard variables used to control for unexplained 
phenomena in this type of study) as substitutes that captured this and 
other omitted control variables. Furthermore, researchers believed that 
including non-funded agencies would bias the results in favor of 
finding an impact of COPS funding on crime.


GAO Audit Overlooks Significant Variable:

The GAO is mistaken in stating that the researchers did not conduct the 
analysis using city dummy variables. In fact, the researchers did 
conduct a separate analysis using city dummy variables in order to 
verify the reported county dummy variable analysis as is indicated in 
footnote 25 of the researchers' study. Use of the city dummy variables 
addresses the GAO issue regarding misspecification of the model - a 
large part of the GAO's critique. In addition, the inclusion of these 
variables attempts to compensate for the absent control variables that 
the GAO mentions.

Validity of Researchers' Study:

The GAO itself commends the researchers for the considerable efforts 
made to consider alternative evaluation methods and to ensure the 
reliability of the results. Given the recognition by the GAO of the 
limitations inherent in this type of social science research, as well 
as the particularly extensive peer review, the COPS Office is satisfied 
that the study used a sound methodology given the data available at 
that time.

According to the findings of the study, $100,000 in COPS hiring grant 
dollars provided to a city of 100,000 will result in an approximate 
reduction of five violent crimes and twenty-one property crimes, a 
total reduction of twenty-six crimes. Given that the COPS Office has 
provided billions of dollars in funding to cities, the number of 
serious violent and property crimes reduced by COPS grants, according 
to the study, is in the many hundreds of thousands.

Pending Updated Report:

Prior to the GAO report, the COPS Office requested that the researchers 
update the study. The updated study will address the issues raised by 
the GAO and will include: analyzing the 2000 census data not available 
at the time of the initial study, adding city dummy variables as:

was done in the initial study, conducting a separate analysis of the 
possible influence of police expenditures, and adding the non-funded 
agencies in the analysis. We look forward to the results of the updated 
study.

Sincerely,


Pamela Cammarata:

Signed by Pamela Cammarata:

Acting Deputy Director:

Vickie Sloan:

Director, Audit Liaison Office Justice Management Division:

Cynthia A. Bowie Assistant Director COPS Compliance Division:

[End of section]

Enclosure II:

CRIMINAL JUSTICE:

UNIVERSITY OF Omaha:

June 5, 2003:

Laurie Ekstrand, Director 
Homeland Security and Justice 
United States General Accounting Office:

Washington, DC 20548:

RE: Response to Technical Assessment of Zhao and Thurman's 2001 
Evaluation of the Effects of COPS Grants on Crime:

Dear Ms. Ekstrand,

Our response to the General Accounting Office's (GAO) review of our 
research is as follows. We begin with prefacing remarks that address 
our concerns with the review process, followed by detailed responses to 
specific points of criticism that the GAO review raised regarding the 
quality of our work.

First, we want to apologize upfront for the brevity of our remarks. 
Since the GAO was given such a short timeframe in which to complete 
their review and we were only given 36 hours in which to respond to it 
once the GAO report became available, we feel that we may be 
inadequately prepared in this endeavor. While we initially were hopeful 
that additional scrutiny of our work might shed light on its value as 
the most comprehensive study to date on a very challenging research 
question, we since have been largely disappointed in the approach the 
GAO has taken in its review. But let me further explain.

While we were not surprised by the questions the GAO review team asked, 
we are a bit perplexed at the questions they did not ask and as to why 
the answers that we gave during our teleconference with the GAO review 
team were not later considered in the GAO report (we will elaborate on 
these in more specific detail below).

Harvard University Professor Mark Moore (Moore is also the Chair of the 
Kennedy School's Program in Criminal Justice Policy and Management) 
refers to the work based on the study published in Crime and Public 
Policy (in the same issue of the journal) as something that "could be 
viewed as a classic piece of program evaluation" (p. 36). And while he 
does not fully endorse our approach, he comments that he instead "will 
stand aside in awe of the brute empiricism of a sample of five years of 
federal funding for police and crime experience in 6,100 cities and 
towns" (p.36). He instead levels his criticism at the lack of focus on 
the policy implications of the research and goes on to make a 
persuasive argument that social science research is ill-equipped to 
address policy questions, stating "social science findings can never 
fully dictate the right answer to an important policy question. They 
cannot do this 
even when the methods are deployed powerfully in program evaluations. 
And it is not just because the relevant sciences are not yet mature. It 
is because important normative questions remain entirely beyond the 
reach of science, and because any important policy choice involves 
important positive issues that science has not yet, or could not easily 
ever reach." (p.42). Such is the case here.

Rather, we attempted simply to look at the Office of Community Oriented 
Policing Services' (COPS) support of American policing over time using 
available data sets of a secondary nature and at the aggregate level to 
explore any effects on crime that might be linked to three different 
types of programs that the COPS Office had employed over a six year 
period. This study was not intended to be definitive. We undertook this 
research primarily motivated by intellectual curiosity as we collected 
our data and reported our findings along the course of ongoing analyses 
(and our analyses are still ongoing). Thus, we believe that the GAO 
review team never asked the most compelling question that should have 
been asked, namely how does this work compare with or advance existing 
knowledge on this subject?

Granted, our work may not be perfect (the fact that available data 
needed to do this work had to be fully assembled from scratch and did 
not previously exist in a readily usable form that would have made a 
perfect study possible or even plausible) but we strongly believe it to 
be better and more comprehensive than any previous research on this 
subject. Not only was our work not compared to any other study on this 
topic, the standard applied to our work by the GAO review team was the 
toughest of all-the phantom perfect design standard-which leads us to 
our final remark before getting into more specific detail.

The GAO review failed to consider the value of our work in a relative 
sense. That is, although this study may be imperfect (we would not 
argue this point), the questions that they should ask concern the 
extent to which our analyses are reasonable considering the data that 
exist in the real world with which to explore this research question. 
The GAO is silent on this point unlike the numerous reviews we received 
from knowledgeable and independent sources that were called upon to 
examine this work during a rigorous peer review process. Now on to our 
more specific comments.

We appreciate this opportunity to respond to the issues brought up by 
the General Accounting Office in reference to our research report. The 
GAO presents reasonable questions that were not unanticipated given the 
rigorous peer reviews that we have encountered previously. We have 
carefully weighed the costs and benefits of alternative choices and 
have presented a statistical model that we feel provides the most 
accurate and fair picture of the issue given the available data. We 
would like to address how we dealt with each of these issues 
separately:

Omitted variables:

Ideally, a researcher would like to include the variables of interest 
in the analytic equation to say with absolute confidence that every 
potential control variable is included. In reality, however, there will 
always be variables that should be included but are unavailable for 
inclusion. This issue, omitted variables, is a weaknesses in all social 
science research. There is no exception for the research that we report 
here. Several control variables were simply 
unavailable at the city level. Researchers do their best to address the 
issues using alternate variables and complex statistical modeling. It is 
our opinion that this report does a better job than most at handling 
this ever-present issue.

The GAO report states that the following important variables were 
omitted from the analysis, police expenditure, access to other grant 
programs, a measure of city size, percent male, and per capita income. 
A special statistical advantage of panel data analysis (analysis that 
tracks variables over a period of time) is that it is able to capture 
unobserved variance. By using dichotomous variables (e.g., city, county, 
and year dummy variables), we were able to control for systematic and 
unobserved variance across the time period studied (a period of six 
years). The effect of the county dummy variables, in fact, represents 
the most significant effect in the entire equation. In simple terms, 
the inclusion of these dummy variables is an attempt to statistically 
account for systematic and unobserved control variables that were not 
specifically included in the model. This is the unique benefit of this 
type of panel study as much social science research involves the 
analysis of data at only one point in time (cross-sectional) and is 
unable to use this advanced statistical technique. It should be 
emphasized that the models run in this study (as is noted in footnote 
25) were in fact also run using city dummy variables. This use of city 
dummy variables is an attempt to control for all of the omitted control 
variables mentioned above. In addition, the explained variance in the 
models is at an extremely high level (the R2 of the models ranges from 
.64 to .86). The vast majority of social science research is only able 
to explain a much smaller percentage of the variance (often ranging 
from .10 to .20). Because so much of the variance is explained by the 
dummy variables, this makes it even less likely that the additional 
controls requested would meaningfully affect the results as their 
effects are likely encompassed in the already present controls. 
However, we will address each of the control variables in turn.

Police expenditures:

There is no documented information on police budgets for the agencies 
included in the analysis that is collected annually across the study 
period. Accordingly, as a means of addressing this concern, we adjusted 
the statistical modeling and included the 1994 crime rate and county 
and city dummy variables as substitutes in order to attempt to capture 
this (and other) omitted control variables. In our analysis, the 1994 
crime rate was used to control for the level of crime (presented as a 
rate) in a given city when COPS grants became available. We did this 
based on the assumption that the 1994 crime rate reflected the ability 
of police agencies to control the level of crime incidents by local 
police agencies with given, or "typical," resources including budget, 
personnel, etc. (budgets without COPS funding). In addition, we have 
recently completed a follow-up study in which we collected original 
police expenditure data (minus COPS funding) from 55 of the largest 
police departments. We included this data in statistical models very 
similar to those reported here. The findings showed that the inclusion 
of police expenditure had virtually no effect on the COPS funding. 
Because of this, and because an attempt was made to take this variable 
into account in the above mentioned ways, we find it unlikely that the 
inclusion of this additional control variable would alter the findings.

Access to other types of grants:

The GAO report raises the possibility that some agencies are more adept 
at receiving all kinds of grant funding, including COPS grants, and 
that this may bias the results in someway. First, this data would be 
very difficult to obtain, particularly at the agency level distributed 
by each state across the years of study. Second, there is an attempt to 
control for factors such as this in the model through the use of county 
and alternatively city dummy variables as is discussed above. Moreover, 
the COPS Office has provided funding to over 12,000 of the 
approximately 18,000 law enforcement agencies in the country. In 
addition, the COPS grant application process has been widely regarded 
by law enforcement as one of the most user-friendly. Thus, it appears 
that the application process for COPS grants is such that a very wide 
range of agencies has access to them. Because of the widespread access 
to COPS grants and because of the inclusion of the dummy variables we 
felt it unlikely that this possibility would alter the reported results 
and did not have any other means by which to take this variable into 
account.

Measure of city size:

City size is standardized in the models through the examination of 
crime rates. In addition, all the models are weighted by city 
population. We did determine that there may be an interaction effect 
between city size and the effect of COPS grants on crime. Therefore, we 
did conduct an examination of this by splitting the model into two 
categories, those with populations greater than 10,000 and those with 
populations less than 10,000. The overall crime drop in the 1990s was 
dramatically apparent across the nation. However, the rate of the crime 
drop differed significantly among cities. One of the most important 
variable to distinguish the variation in the patterns of this crime 
drop is the size of the city. The crime drop, for example, was more 
closely associated with larger cities than with smaller ones as is 
detailed in the study. In our analysis, we found that the crime drop 
was indeed different between larger and smaller cities. We decided, 
consequently, to adopt the method used in the Uniformed Crime Report 
compiled by the FBI. Following their lead, we decided to split the 
police agencies into two categories: cities with a population of 10,000 
and over versus cities with a population of 10,000 and below. This 
design allowed us to uncover the differences between the effect of COPS 
funding between these two groups.

Socioeconomic Variables:

The GAO report suggests that per capita income and percent male should 
also be included in the models. However, these effects, if any, would 
be captured through the inclusion of other socioeconomic variables and 
the city dummy variables as mentioned above. Moreover, another 
important issue of omitted variables concerns the relationship between 
the specific omitted variables and COPS grants. If the relationship 
between the two is orthogonal (little association) then the exclusion 
of that omitted variable won't introduce bias in estimating the effect 
of COPS grants. For example, there is no evidence to suggest that 
decisions made by the COPS Office to distribute grants or for agencies 
to acquire grants is in any way based on percentage of male in a city. 
Then, the percentage of male will not significantly affect the 
contribution of COPS grants on crime reduction. Furthermore, 
multiconearity is always a:

problem when socioeconomic variables are used in an analysis. 
Therefore, social scientists attempt to select a few crucial ones 
instead of casting a big net. In our analysis, the six socioeconomic 
variables (percentage of unemployment, percentage of minority, 
percentage of single mother household, percentage of young people 
between 15 to 24, percentage of living in the same home for the past 
five years, and percentage of home owners) are derived from social 
disorganization theory developed by Shaw and McKay and have been widely 
used for 
the past fifty years. These variables are used in the studies published 
in the best journals of our discipline (e.g., Osgood and Chambers 2000; 
Reisig and Parks 2000). In contrast, we don't think that researchers 
have ever used percentage of male in a similar analysis. In 
addition, the percentage of male is relatively stable over the last 
decade. There is no reason to believe that the male population increased 
sharply during the past decade in the U.S. with a similar sharp decline 
of female population. Almost certainly not enough to account for any of 
the relationship between COPS grants and crime.

Misspecifications in the analytic models:

We were fully aware of the issues involved with the use of county dummy 
variables, not city dummies in our 2001 study. We decided to use county 
dummy because the fixed effect of city dummy would wipe out all the 
contribution of 1990 socioeconomic variables. In our follow-up study, 
both county and city dummies were used when socioeconomic variables 
were time-varying across the years of study. The findings are 
consistent in that the use of county and city dummies was not a 
decisive factor in determining the effect of COPS grant on crime 
reduction between 1995 and 2000.

Therefore, the results of a follow-up study on the use of city and 
county dummies should be incorporated into this section, and it should 
be made clear that there is no misspecification in the analytic models 
in the follow-up study. Further, the comparison between the two studies 
indicates that county dummies do not introduce significant estimation 
bias in the 2001 study. In addition, as is indicated in footnote 25, 
the city dummy variable analysis of the initial study was conducted in 
an attempt to further verify the results.

Sample selection is limited:

The missing cases in the model are due to two factors. First, agencies 
with populations less than 1,000 were removed. As is described in the 
report, these agencies were removed because they introduced a great 
deal of variation in the model that made the estimates unstable. The 
other factor was missing UCR data. A few states (Delaware, Kansas and 
Illinois) do not typically report UCR data to the FBI. We are well aware 
of this limitation.

The report is clear that it only addresses COPS grants to cities and 
does not evaluate the effect on grants to special police, county, state 
and university police. There is no meaningful way to include these 
types of agencies in the model because of overlapping jurisdictions. If 
they were to be included this would introduce a substantial amount of 
error into the 
analytical models. Thus, the analysis was limited to the defined 
boundaries of cities and both the COPS and crime variables were 
measured at this level. Furthermore, the GAO study argued that nested 
effect of other law enforcement agencies located in the city (e.g.. 
school 
district police, university police, park police, sheriff office, etc) 
may also contribute to the crime decline in a city (usually big 
cities). Theoretically, they are right, but practically it is not 
necessary and impossible to estimate the effect of these agencies' 
contribution due to two primary reasons. First, these agencies report 
their own separate annual crime incidents to the FBI. It is reasonable 
to assume that any crossover due to reporting is minimal. Second, there 
is no census information on school district police, park police, etc. 
Lack of socioeconomic variables makes the estimate of the effect of 
these agencies impossible. In the hundreds of studies on city crime in 
the past four decades we don't recall even a single study that attempts 
to measure the crossover by including school district police, court 
police, park police in an analysis.

Including non funded agencies:

In an ideal analysis, a researcher will analyze differences between two 
groups: an experimental group and a control group. Additionally, the 
assignment of individuals to each of these groups should be random, in 
order to make reliable comparisons in outcomes between the groups. In 
social science research, however, a design of this nature is rarely 
utilized and often not possible. This is the case in the current study.

In the final data set, all cities with greater than 150,000 population 
received COPS grants between 1994 and 1999 (the period of study). 
Additionally, about 90% of cities greater than 10,000 population were 
funded by the COPS Office during the same period. Consequently. this 
analysis not only includes the population of all the large cities in 
the United States (with the exception of Chicago due to missing UCR 
data) but also a sample of many much smaller cities that received COPS 
funding as well. Since the COPS Office has provided funding to the vast 
majority of municipal city police departments, the numbers of non-
funded agencies is small in the cities with a population over 10,000. 
For example, in 1994, there were only 335 non-funded cities over 10,000 
population that were excluded from the analysis (there were over 2,200 
COPS funded cities in 1994 in the same population group with a 
selection of 12 months reported). The mean population of these cities 
in 1994 was 29,675. Thus, the inclusion of these 335 cities is unlikely 
to influence the significant results found in this population category 
as there are so few of them.

Moreover, we believe that these non-funded agencies in cities with a 
population over 10,000 do not represent a similar comparison group. 
That is, if a comparison group of non-funded agencies were to be 
included, it may bias the findings towards showing an effect of COPS 
grants because these non-funded agencies would be relatively small in 
size.

Our design, a panel data analysis, allows us to assess the effect of 
COPS funding between agencies that received a large amount of grant 
funding with those that received a small amount or no funding in a 
given year during the period of study. This type of analysis allows for 
an investigation into whether variation in funding would lead to a 
variation in the crime drop between 1995 and 2000. This type of 
comparison of variation in "treatment" levels (funding amounts in this 
case) is at the very heart of much social science research. However, we 
will conduct additional analysis in the follow-up study to include non-
funded agencies 
with complete UCR data and the results will be put in the Appendix 
section of the updated report.

Summary:

In total, the COPS Office sent this and the similar follow-up study to 
six anonymous independent external peer reviewers. In addition, a 
version of this study has been published in Criminology and Public 
Policy, a well-respected journal sponsored by the American Society of 
Criminology. In order to gain admittance to this journal, the paper 
underwent an additional thorough peer review process incorporating the 
comments of five additional separate professional reviewers. The 
successful publication in this journal speaks to the soundness of the 
methodology used in comparison with other published research projects.

We would welcome specific recommendations from the GAO team outlining 
their vision of a practical study design that can be conducted in 
reality. In addition, we would like to discuss with them several 
specific issues involved with measurement (e.g., how one would account 
for the contribution of campus police or citizen participation at the 
city level).

All social scientists are confronted with a variety of methodological 
choices when conducting research. We have carefully weighed the costs 
and benefits of alternative choices and have presented a statistical 
model that we feel provides the most accurate and fair picture of the 
issue given the available data. Because no piece of social science 
research is perfect it should always be viewed as adding to the body of 
knowledge with respect to a specific topic. We feel that this research 
accomplishes this goal as it provides some evidence that is on par with 
and in some ways superior to the types of evidence typically referenced 
regarding these types of policy relevant issues.

Jihong "Solomon" Zhao, Ph.D. 

University of Nebraska at Omaha 


Quint Thurman, Ph.D. 
Southwest Texas State University:

Signed by Jihong "Solomon" Zhao, Ph.D.:


FOOTNOTES

[1] Zhao, J. and Thurman, Q. A National Evaluation of the Effect of 
COPS Grants on Crime from 1994 to 1999 (Dec. 2001).

[2] Zhao, J., Scheider, C. and Thurman, Q. Funding Community Policing 
to Reduce Crime: Have COPS Grants Made a Difference? Journal of 
Criminology & Public Policy, Nov. 2002 (vol. 2, no. 1).

[3] P.L. 103-322.

[4] Statistical significance means that the observed effect does not 
result from chance alone. The number of observations in a sample can be 
an important determinant of statistical significance, with larger 
sample sizes frequently being associated with statistically significant 
findings. Zhao et al.'s 2001 study consisted of 36,605 observations, 
making it possible that statistically significant effects could have 
been found even when they were small on a practical level. 

[5] This is calculated as follows: From table 2, we see that in cities 
larger than 10,000, each dollar of innovative grant funding was 
associated with a decrease of 12.93 violent crimes. $0.42 is 42 percent 
of 1 dollar, and 42 percent of 12.93 crimes equals 5.4. This represents 
the decrease in the expected crime rate as innovative grant funding 
increased by $0.42 per person. If the violent crime rate were 770 per 
100,000 population, doubling the $0.42 innovative grant expenditure per 
person would reduce the violent crime rate by 5, or to about 765 per 
100,000 population.

[6] The mean offered in Zhao and Thurman is a weighted average for all 
cities and only approximates the mean for large cities. Because of 
that, and the severe skew in the distribution of average grant amounts 
across cities (note the standard deviations in table 1), this may not 
be a very accurate way to estimate the effect size. The skew in the 
distribution of grant amounts also suggests that it might have been 
preferable to transform (using logarithms) those amounts prior to the 
analyses. 

[7] A footnote in the 2001 study indicates that the researchers 
conducted initial analyses using city dummy variables. However, they 
ultimately decided to use county dummy variables, and all the report 
findings are derived from statistical models that included county 
rather than city dummy variables.

[8] For example, if the city of College Park, MD, received a COPS grant 
and the University of Maryland campus police (located in College Park) 
received a separate COPS grant, their joint impact on the city's crime 
rates would not be included in this analysis. 

[9] In the analysis, the crime rates from 1995 to 1999 were 
intentionally lagged a year to allow these agencies to receive and 
deploy these funds.

[10] UCR is a nationwide database of police statistics consisting of 
crime data voluntarily reported to the Federal Bureau of Investigation 
by nearly 17,000 city, county, and state law enforcement agencies. UCR 
data form the basis for a Crime Index, which is used to gauge 
fluctuations in the nation's overall volume and rate of crime. The 
offenses included in the "violent crime" category are murder and 
nonnegligent manslaughter, forcible rape, robbery, and aggravated 
assault. The offenses included in the "property crime" category are 
burglary, larceny-theft, and motor vehicle theft, and arson. 

[11] In personal discussions with the researchers, we learned that 
their 2001 published study contained an error related to missing data. 
Specifically, the researchers had intended to eliminate cities from 
their analysis if crime data were missing for even a single month of 
the year. However, the dataset they obtained did not uniformly 
distinguish between missing data and "zero" reported crimes. In those 
cases, the analysis would have produced an underestimate of the 12-
month crime rate. After publishing their results, the researchers 
corrected these data errors and reanalyzed the dataset. They told us 
that the revised results did not differ substantially from those 
published. Time limitations prevented us from assessing the revised 
results.

[12] In a November 2002 publication in the Journal of Criminology and 
Public Policy, Zhao et al. explained that their analyses omitted cities 
without COPS grants because of concern that including these cities 
would produce a downward bias in their estimation of COPS program 
effects. They said this is because crime was decreasing across the 
board between 1994 and 1998 in cities with and without COPS grants. We 
disagree with their rationale. Since Zhao and Thurman controlled for 
the baseline rate of crime by including the 1994 rate in their model as 
a control variable, cities with COPS grants would presumably have a 
higher rate of decrease than cities without COPS grants. We continue to 
believe that Zhao and Thurman's estimates of COPS program effects were 
biased as a result of omitting cities that did not receive COPS grants.

[13] One difference in the crime rates analyzed in the two studies was 
that arson was included as a property crime in the newer study, but not 
in the 2001 study.

[14] The authors provided us with additional information from their 
follow-up study on the analytic results obtained when they used dummy 
variables to represent cities instead of counties. They found that in 
large cities, the estimated effects of hiring grants on violent crimes 
doubled, the estimated effects of MORE grants doubled and became 
statistically significant, and the effect of innovative grants became 
statistically not significant. The effect of MORE grants on property 
crimes remained significant in large cities when city dummy variables 
were used, but diminished to half the size that was estimated by a 
model that used county dummy variables. 

[15] The researchers noted that they recently collected original police 
expenditure data from 55 of the largest police departments and found 
that including these data in the statistical models showed that they 
had virtually no effect on their estimates of the effects of COPS 
funding. We appreciate the difficulty of obtaining police expenditure 
data for large cities and endorse efforts to marshal supporting 
evidence from a sample of those cities. However, we have not seen the 
results of these analyses and have no basis to judge how representative 
these 55 cities are of large cities in general, or whether the 
estimated effects of COPS grants from the 55 cities are generalizable 
to larger cities generally.

[16] Bureau of Justice Statistics data on 62 local police departments 
serving cities with a population of 250,000 or more revealed a high 
degree of change in violent and property crime rates within the same 
city over time. For example, New York's reported violent crime rate 
dropped by 57 percent between 1990 and 2000, while Nashville's rate 
increased by 29 percent during that same time period. Similarly, New 
York's reported property crime rate dropped by 60 percent, while 
Nashville's rate increased by 22 percent. (Police Departments in Large 
Cities, 1990-2000. Department of Justice, Bureau of Justice Statistics, 
May 2002).

[17] Multicollinearity means that the independent variables are highly 
correlated. If this occurs, it is impossible to distinguish between 
them in estimating their effects on the dependent variable.

[18] For example, a Bureau of Justice Statistics study indicated that 
men comprised 93 of the state prison population in 2001; 93 percent of 
the federal prison population in 1997; and 90 percent of the local jail 
population in 1996 (http://www.ojp.usdoj.gov/bjs/
crimoff.htm#findings). Another study reported that in 1960, 1975, and 
1990, men were arrested at much higher rates than women for all crime 
categories except prostitution (Steffenmeier, D. and Allen, E., "Gender 
and Crime: Toward a Gendered Theory of Female Offending," Annual Review 
of Sociology, 1996, vol. 22, pp. 459-87).