This is the accessible text file for GAO report number GAO-05-71 
entitled 'Census 2000: Design Choices Contributed to Inaccuracy of 
Coverage Evaluation Estimates' which was released on December 13, 2004.

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as part 
of a longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov.

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately.

Report to Congressional Requesters: 

November 2004: 

CENSUS 2000: 

Design Choices Contributed to Inaccuracy of Coverage Evaluation 
Estimates: 

GAO-05-71: 

GAO Highlights: 

Highlights of GAO-05-71, a report to congressional requesters

Why GAO Did This Study: 

Evaluations of past censuses show that certain groups were undercounted 
compared to other groups, a problem known as “coverage error.” To 
address this, the Census Bureau included in its 2000 Census design the 
Accuracy and Coverage Evaluation Program (A.C.E.) to (1) measure 
coverage error and (2) use the results to adjust the census, if 
warranted. However, the Bureau found the A.C.E. results inaccurate and 
decided not to adjust or plan for adjustment in 2010.

Congress asked GAO to determine (1) factors contributing to A.C.E.’s 
reported failure to accurately estimate census coverage error, and (2) 
the reliability of the revised coverage error estimates the Bureau 
subsequently produced.

To do this, GAO examined three sets of Bureau research published in 
March 2001, October 2001, and March 2003 and interviewed Bureau 
officials.


What GAO Found: 

According to senior Bureau officials, increasingly complicated social
factors, such as extended families and population mobility, presented 
challenges for A.C.E., making it difficult to determine exactly where 
certain individuals should have been counted thus contributing to the 
inaccuracy of the coverage error estimates. For example, parents in 
custody disputes both may have an incentive to claim their child as a 
resident, but the Bureau used rules for determining where people should 
be counted–residence rules--that did not account for many of these 
kinds of circumstances. Other design decisions concerning both A.C.E. 
and the census also may have created “blind spots” that contributed to 
the inaccuracy of the estimates (see figure). The Bureau has not 
accounted for the effects of these or other key design decisions on the 
coverage error estimates, which could hamper the Bureau’s efforts to 
craft a program that better measures coverage error for the next 
national census.

Factors Potentially Affecting Accuracy of Coverage Error Estimates at 
Different Points in the A.C.E. Program: 

[See PDF for image]

[End of figure]


Despite having twice revised A.C.E.’s original coverage error 
estimates, the Bureau has no reliable estimates of the extent of 
coverage error for the 2000 census. While both revisions suggested that 
the original estimates were inaccurate, in the course of thoroughly 
reviewing the revisions, the Bureau documented (1) extensive 
limitations in the revision methodology and (2) an unexpected pattern 
between the revised estimates and other A.C.E. data, both of which 
indicated that the revised coverage error estimates may be questionable 
themselves. Furthermore, when the Bureau published the revised 
estimates, it did not clearly quantify the impact of these limitations 
for readers, thus preventing readers from accurately judging the 
overall reliability of the estimates. It is therefore unclear how 
A.C.E. information will be useful to the public or policymakers, or how 
the Bureau can use it to make better decisions in the future.

What GAO Recommends: 

GAO is making recommendations related to how the Bureau handles design 
decisions and reports results to improve planning and reporting of 2010 
census and coverage evaluation. 

The Department of Commerce concurred with the recommendations but 
raised several concerns regarding our analyses, which we have discussed 
in the report and used to revise it. 

www.gao.gov/cgi-bin/getrpt?GAO-05-71.

To view the full product, including the scope and methodology, click on 
the link above. For more information, contact Patricia A. Dalton at 
(202) 512-6806 or daltonp@gao.gov.

[End of section]

Contents: 

Letter: 

Results in Brief: 

Background: 

Scope and Methodology: 

Design and Other Bureau Decisions Created Difficulties and Blind Spots 
in Census Coverage Evaluation: 

The Bureau Has Not Produced Reliable Revised Estimates of Coverage 
Error for the 2000 Census: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments and Our Evaluation: 

Appendixes: 

Appendix I: Comments from the Department of Commerce: 

Appendix II: Bureau Design Decisions Increased Imputations: 

Appendix III: Bureau Estimates of Population and Housing Unit Net 
Undercounts: 

Glossary: 

Related GAO Products: 

Tables: 

Table 1: Estimated Percent Population Undercounts for Selected Race/
Origin Groups: 

Table 2: Estimated Percent Occupied Housing Undercounts Differ from 
Population Undercounts for Selected Race/Origin Groups: 

Figures: 

Figure 1: A.C.E. Sample Operations Paralleled the Census: 

Figure 2: A.C.E. Sample Excluded Individuals and Records from Scope 
of Evaluation: 

Figure 3: A.C.E. Matching Results Lacked Data on Suspected Duplicates 
That Were in the Census: 

Figure 4: Reported Undercount Estimates Generally Decreased with A.C.E. 
Revisions: 

Figure 5: Population and Housing Undercounts: 

Abbreviations: 

A.C.E.: Accuracy and Coverage Evaluation: 

NAS: National Academy of Sciences: 

Letter November 12, 2004: 

The Honorable Henry A. Waxman: 
Ranking Minority Member: 
Committee on Government Reform: 
House of Representatives: 

The Honorable Danny K. Davis: 
Ranking Minority Member: 
Subcommittee on Civil Service and Agency Organization: 
Committee on Government Reform: 
House of Representatives: 

The Honorable Wm. Lacy Clay, Jr.: 
Ranking Minority Member: 
Subcommittee on Technology, Information Policy, Intergovernmental 
Relations and the Census: 
Committee on Government Reform:
House of Representatives: 

The Honorable Charles A. Gonzalez: 
The Honorable Carolyn B. Maloney: 
House of Representatives: 

A decennial census must be as accurate as possible, because census 
results are used to, among other purposes, apportion congressional 
seats, redraw congressional districts, and allocate federal aid to 
state and local governments. However, given the nation's size and 
demographic complexity, some amount of error is inevitable. 
Unfortunately, evaluations of past censuses have shown that certain 
groups, for example African-Americans and Hispanics, have been 
undercounted in comparison to other groups. To estimate the extent that 
some groups were over-or undercounted in 2000--what the Bureau refers 
to as "coverage error"--the Census Bureau (Bureau) planned and 
implemented the Accuracy and Coverage Evaluation (A.C.E.) program. The 
primary goals of A.C.E. were to: 

* more accurately estimate the rate of coverage error via a sample 
survey of select areas nationwide, and: 

* if warranted, to use the results of this survey to adjust census 
estimates of the population for nonapportionment purposes.

In March 2003, after much deliberation and research, the Bureau decided 
not to use any A.C.E. estimates of coverage error to adjust the 2000 
Census, because it judged these estimates no more accurate than the 
official census data. The Bureau found that A.C.E. did not account for 
at least 3 million erroneously counted persons (mostly duplicates, or 
people counted twice) in the census, which raised questions about the 
reliability of the coverage error estimates. Furthermore, because of 
the difficulties the Bureau experienced in trying to produce reliable 
coverage error estimates, it announced officially in January 2004 that 
it did not plan to develop a procedure for adjusting the 2010 Census 
results for redistricting purposes. Agency officials said that in light 
of their past experiences, they do not think they can produce reliable 
coverage error estimates in time to meet deadlines for adjusting the 
census.

This report responds to your request that we review why A.C.E. coverage 
error estimates were reportedly not sufficiently reliable to adjust or 
validate the 2000 Census. Specifically, our review examined (1) factors 
contributing to A.C.E.'s reported failure to accurately estimate census 
coverage error, and (2) the reliability of the revised coverage error 
estimates the Bureau subsequently produced.

To meet the objectives of this report, we reviewed and analyzed the 
Bureau's publicly available research data and reports on the 2000 
Census. We also reviewed methodology documents and other available 
information such as the minutes and supporting documents of the 
Executive Steering Committees for Adjustment Policy. Finally, we 
discussed the results of our analysis with senior Bureau officials and 
interviewed Bureau officials and committee members to obtain their 
views on the process. Since our focus was on the process and decisions 
that led to the results rather than on determining the underlying 
numbers themselves, we did not audit the Bureau's research, the 
underlying data, or its conclusions. Our work was performed in 
Washington D.C. and at the U.S. Census Bureau headquarters in Suitland, 
Maryland, from December 2002 through July 2004 in accordance with 
generally accepted government auditing standards.

Results in Brief: 

The Bureau attributes the inaccuracy of A.C.E. estimates primarily to 
the rules it used for determining where people should be counted--
residence rules--which officials say did not fully capture the 
increasing complexity of American society. For example, parents in 
custody disputes both may have an incentive to claim their child as a 
member of their respective household, but the Bureau's residence rules 
did not always account for these kinds of circumstances, thus 
accurately counting such individuals was difficult. While the Bureau 
emphasizes residence rules, our research indicates that some of the 
Bureau's other design decisions concerning both A.C.E. and the census 
created several "blind spots" in A.C.E. that also may have contributed 
to the unreliability of coverage error estimates. For example, the 
Bureau decided to exclude from A.C.E. people living in group quarters 
such as college dorms, because their high mobility made them difficult 
to count during the 1990 coverage evaluation, and this weakened the 
1990 coverage error estimates. In addition, the Bureau removed 
2.4 million records it suspected were duplicates from the 
census population before A.C.E. had a chance to measure them, and then 
reinstated them after A.C.E. was finished, so that A.C.E. was blind to 
over 1 million duplicates (according to Bureau research) in that 
reinstated group. However, because the Bureau has not accounted for and 
clearly reported how these and other key design decisions affected the 
outcome of A.C.E., the full range of reasons why its estimates were 
unreliable remains obscure. This in turn could hamper the Bureau's 
efforts to craft a more successful coverage measurement program that 
better measures the accuracy of the next national census.

The Bureau's revised coverage error estimates cannot necessarily be 
interpreted as any more reliable than the original estimates. In the 
course of an extensive review of A.C.E.'s results over a 3-year period, 
the Bureau revised its estimates of A.C.E. coverage error twice--first 
in October 2001 and second in March 2003--and both revisions suggested 
that the original estimates were inaccurate. However, in reviewing the 
revised estimates, the Bureau documented (1) extensive limitations in 
the revision methodology and (2) an unexpected pattern between the 
revised estimates and other data, both of which indicated that the 
revised estimates may be questionable themselves. Furthermore, while 
the Bureau described the aforementioned limitations in detail in its 
review documents, when it published the revised estimates of coverage 
error, it did not clearly quantify their impact for readers. It is 
therefore unclear how these revised estimates will be useful to the 
public or policymakers, or how the Bureau will use them to make better 
decisions in the future.

As a result, we are recommending that the Bureau's future evaluation 
planning take into account the potential effect of future decisions 
relating to census or coverage evaluation design to avoid similar or 
other problems in the 2010 measure of coverage error. We also recommend 
that the Bureau clearly report how any significant changes in the 
design of the census and/or A.C.E. might affect the accuracy of the 
published coverage error estimates for 2010. Similarly, GAO recommends 
that the Bureau not only identify methodological limitations--as it 
currently does--but also report the potential range of impacts that 
methodological limitations and design changes may have on the census 
coverage error estimates it publishes in the future.

The Under Secretary for Economic Affairs forwarded written comments 
from the Department of Commerce on a draft of this report (see app. I). 
The Department concurred with our recommendations but took exception to 
several statements in our draft report, writing that it thought some of 
our conclusions were misleading. In response, we revised the report, as 
appropriate, including additional information provided by the 
Department and clarifying the presentation of our analyses. We discuss 
the Department's comments more fully in the section "Agency Comments 
and Our Evaluation." 

Background: 

Since 1980, the Bureau has used statistical methods to generate 
detailed estimates of census undercounts and overcounts, including 
those of particular ethnic, racial, and other groups. To carry out the 
2000 Census's Accuracy and Coverage Evaluation program (A.C.E.), the 
Bureau conducted a separate and independent sample survey that, when 
matched to the census data, was to enable the Bureau to use statistical 
estimates of net coverage errors to adjust final census tabulations 
according to the measured undercounts, if necessary. The Bureau 
obligated about $207 million to its coverage evaluation program from 
fiscal years 1996 through 2001, which was about 3 percent of the $6.5 
billion total estimated cost of the 2000 Census.

While the A.C.E. sample survey of people was conducted several weeks 
after Census Day, April 1, the "as of" date on which the total 
population is to be counted, many of the processes were the same as the 
2000 Census. For the census, the Bureau tried to count everybody in the 
nation, regardless of their dwelling, and certain kinds of dwellings, 
including single-family homes, apartments, and mobile homes, along with 
demographic information on the inhabitants. For A.C.E., the Bureau 
surveyed about 314,000 housing units in a representative sample of 
"clusters"--geographic areas each with about 30 housing units. The 
sample comprised roughly 12,000 of the about 3 million "clusters" 
nationwide.

As illustrated in figure 1, the Bureau used a similar process to 
develop address lists, collect response data, and tabulate and 
disseminate data--one for the decennial census and one for A.C.E. 
sample areas. For the Census, the Bureau mailed out forms for mail-back 
to most of the housing units in the country; hand-delivered mail-back 
forms to most of the rest of the country; and then carried out a number 
of follow-up operations designed to count nonrespondents and improve 
data quality. A.C.E. collected response data through interviewing from 
April 24 through September 11, 2000.

Figure 1: A.C.E. Sample Operations Paralleled the Census: 

[See PDF for image]

[End of figure]

After the census and A.C.E. data collection operations were completed, 
the Bureau attempted to match each person counted on the A.C.E. list to 
the list of persons counted by the 2000 Census in the A.C.E. sample 
areas to determine exactly which persons had been missed or counted 
more than once by either A.C.E. or the Census. The results of the 
matching process, along with data on the racial/ethnic and other 
characteristics of persons compared, were to provide the basis for 
A.C.E. to estimate the extent of coverage error in the census and 
population subgroups and enable the Bureau to adjust the final 
decennial census tabulations accordingly. The matching process needed 
to be as precise and complete as possible, since A.C.E. collected data 
on only a sample of the nation's population, and small percentages of 
matching errors could significantly affect the estimates of under-and 
overcounts generalized to the entire nation.

Since the 2000 Census, we have issued four other reports on A.C.E., 
addressing its cost and implementation as part of our ongoing series on 
the results of the 2000 Census, as well as a report on the lessons 
learned for planning a more cost-effective census in 2010. (See the 
Related GAO Products section at the end of this report for the 
assessments issued to date.) These reports concluded, among other 
things, that while (1) the address list the Bureau used for the A.C.E. 
program appeared much more accurate than the preliminary lists 
developed for the 2000 Census and (2) quality assurance procedures were 
used in the matching process, certain implementation problems had the 
potential to affect subsequent matching results and thus estimates of 
total census coverage error.[Footnote 1]

In the end, the Bureau decided not to use A.C.E.'s matching process 
results to adjust the 2000 Census. In March 2001, a committee of senior 
career Bureau officials recommended against using A.C.E. estimates of 
census coverage error to adjust final census tabulations for purposes 
of redistricting Congress. In October 2001, the committee also 
recommended against adjusting census data used for allocating federal 
funds and other purposes, largely because Bureau research indicated 
that A.C.E. did not account for at least 3 million erroneously counted 
persons (mostly duplicates) in the census, raising questions about the 
reliability of coverage error estimates. In March 2003, after 
considerable additional research, the Bureau published revised coverage 
error estimates and again decided not to adjust official census data, 
this time for the purpose of estimating the population between the 2000 
and 2010 censuses.

In light of its 2000 experience, the Bureau officially announced in 
January 2004 that while it plans to fully evaluate the accuracy of the 
2010 census, it will not develop plans for using these coverage error 
estimates to adjust the 2010 Decennial Census. Bureau officials have 
told us that there is insufficient time to carry out the necessary 
evaluations of the coverage estimates between census field data 
collection and the Bureau's legally mandated deadline (within 12 months 
of Census Day) for releasing redistricting data to the states. 
Furthermore, the Bureau does not believe adjustment is possible. In 
responding to an earlier GAO report recommending that the Bureau 
"determine the feasibility" of adjusting the 2010 Census, the Bureau 
wrote that the 2000 Census and A.C.E. was "a definitive test of this 
approach" which "provided more than ample evidence that this goal 
cannot be achieved."[Footnote 2] However, in March, the National 
Academy of Sciences (NAS) published a report that recommended that the 
Bureau and the administration request and Congress provide funding for 
an improved coverage evaluation program that could be used as a basis 
for adjusting the census, if warranted.[Footnote 3] The Academy agrees 
with the Bureau that 12 months is insufficient time for evaluation and 
possible adjustment; in the same publication, NAS recommended Congress 
consider extending the statutory deadline of 12 months for providing 
data for redistricting purposes, a suggestion which, if appropriate, 
could make adjustment possible.

Scope and Methodology: 

To identify the factors that may have contributed to A.C.E. missing 
coverage errors in the census, we reviewed evaluations of A.C.E. and 
the Bureau's subsequent revisions to its estimation methodology, as 
well as changes made to the design from its 1990 attempts to estimate 
coverage. We interviewed Bureau officials responsible for A.C.E. 
decision making to obtain further context and clarification. We did not 
attempt to identify all factors contributing to the success or failure 
of A.C.E in estimating coverage error. Since our focus was on the 
process and decisions that led to the results rather than on 
determining the underlying numbers themselves, we did not audit the 
Bureau's research, the underlying data, or its conclusions. We relied 
on the Bureau's own reporting quality assurance processes to assure the 
validity and accuracy of its technical reporting, and thus we did not 
independently test or verify individual Bureau evaluations of their 
methodologies.

To identify the extent of the census errors not accounted for by 
A.C.E., we reviewed the descriptive coverage error estimates and the 
limitations and context of these data as described in the Bureau 
reports published by the Bureau in March 2001, October 2001, and March 
2003.

On, August 9, 2004, we requested comments on the draft of this report 
from the Secretary of Commerce. On September 10, 2004, the Under 
Secretary for Economic Affairs, Department of Commerce forwarded 
written comments from the department (see app. I), which we address in 
the "Agency Comments and Our Evaluation" section at the end of this 
report.

Design and Other Bureau Decisions Created Difficulties and Blind Spots 
in Census Coverage Evaluation: 

The following Bureau decisions concerning the design of the census and 
the A.C.E. program created difficulties and blind spots for the 
coverage evaluation, possibly preventing A.C.E. from reliably measuring 
coverage error: (1) using residence rules that were unable to capture 
the complexity of American society, (2) excluding the group quarters 
population from the A.C.E. sample survey, (3) making various decisions 
that led to an increase in the number of "imputed" records in the 
census, (4) removing 2.4 million suspected duplicate persons from the 
census but not the A.C.E. sample, and (5) reducing the sample area 
wherein A.C.E. searched for duplicates during matching. However, the 
Bureau has not accounted for how these design decisions have affected 
coverage error estimates, which has prevented it from pinpointing what 
went wrong with A.C.E., and this in turn could hamper its efforts to 
craft a more successful coverage measurement program for the next 
national head count.

Bureau Attributes Inaccuracy of Coverage Error Estimates to Residence 
Rules That Did Not Capture Complexity of U.S. Society: 

Bureau officials attribute A.C.E.'s inaccuracy primarily to the fact 
that it used residence rules that do not fully capture the complexity 
of American society. According to senior Bureau officials, increasingly 
complicated social factors, such as extended families and population 
mobility, presented challenges for A.C.E., making it difficult to 
determine exactly where certain individuals should have been counted. 
Specifically, in developing A.C.E. methodology, Bureau officials 
assumed that each person in its sample could be definitively recorded 
at one known residence that the Bureau could determine via a set of 
rules. However, individuals' residency situations are often 
complicated: Bureau officials cite the example of children in custody 
disputes whose separated parents both may have strong incentives to 
claim the children as members of their household, despite census 
residence rules that attempt to resolve which parent should report the 
child(ren). In such situations, wherein the residence rules are either 
not understood, are not followed, or do not otherwise provide 
resolution, the Bureau has difficulty determining the correct location 
to count the children. Bureau officials cite similar difficulties 
counting college students living away from home, as well as people who 
live at multiple locations throughout the year, such as seasonal 
workers or retirees.

A.C.E. design also assumed that follow-up interviews would clarify and 
improve residence data for people for whom vague, incomplete, or 
ambiguous data were provided and whose cases remained unresolved. 
However, the Bureau found it could not always rely on individuals to 
provide more accurate or complete information. In fact, in our earlier 
reporting on A.C.E. matching, we described several situations wherein 
conflicting information had been provided to the Bureau during follow-
up interviews with individuals, and Bureau staff had to decide which 
information to use.[Footnote 4] More recently, the Associate Director 
for Decennial Census told us that returning to an A.C.E. household to 
try and resolve conflicting data sometimes yielded new or more 
information but not necessarily better information or information that 
would resolve the conflict. The Bureau plans to review and revise its 
census residence rules for 2010, which may clarify some of this 
confusion.

Other Design Decisions Also Created Blind Spots in Census Coverage 
Evaluation: 

While the Bureau emphasizes residence rules as the primary cause of 
A.C.E. failure, our research indicates some of the Bureau's other 
design decisions created blind spots that also undermined the program's 
ability to accurately estimate census error. For example, the Bureau 
decided to leave people living in group quarters--such as dormitories 
and nursing homes--out of the A.C.E. sample survey, which effectively 
meant they were left out of the scope of A.C.E. coverage evaluation 
(see fig. 2).

Figure 2: A.C.E. Sample Excluded Individuals and Records from Scope 
of Evaluation: 

[See PDF for image]

Note: Figures are not drawn to proportion.

[End of figure]

As a result, the matching results could not provide coverage error 
information for the national group quarters population of 7.8 million. 
In addition, the Bureau did not design A.C.E. matching to search for 
duplicate records within the subset of this population counted by the 
census, though later Bureau research estimated that if it had, it would 
have measured over 600,000 additional duplicates there. In response to 
our draft report the Department wrote that coverage evaluation was 
designed to measure some of these duplicates since during its follow-up 
interviews at households during A.C.E. matching, the Bureau included 
questions intended to identify college students living away at college.

While coverage evaluation in 1990 included some group quarters, such as 
college dormitories and nursing homes, within its sample, the Bureau 
reported that the high mobility of these people made it more difficult 
to count them, thus the 1990 estimates of coverage for this population 
were weak. The Bureau decided not to gather these data during 2000 
A.C.E. data collection based in part on the difficulty of collecting 
and matching this information in the past, and in part as a result of a 
separate design decision to change the way to treat information for 
people who moved during the time period between the census and the 
coverage evaluation interviews. By excluding group quarters from the 
coverage evaluation sample, the Bureau had less information collected 
on a sample of this population that included some duplication, and the 
missing information may have enabled it to better detect and account 
for such duplication. In addition, by developing coverage error 
estimates that were not applicable to the group quarters population, 
the Bureau made the task of assessing the quality of the census as a 
whole more difficult.

Figure 2 also shows that another blind spot emerged as the Bureau 
increased the number of "imputed"[Footnote 5] records in the final 
census, though they could not be included in the A.C.E. sample survey. 
The Bureau estimates a certain number of individuals--called 
imputations--that they have reason to believe exist, despite the fact 
that they have no personal information on them, and adds records to the 
census (along with certain characteristics such as age and race/
ethnicity) to account for them. For example, when the Bureau believes a 
household is occupied but does not have any information on the number 
of people living there, it will impute the number of people as well as 
their personal characteristics. The Bureau increased imputations in 
2000 from about 2 million in 1990 to about 5.8 million records. Changes 
in census and coverage evaluation design from 1990 likely contributed 
to this increase. Specifically, the Bureau reduced the number of 
persons who could have their personal information recorded on the 
standard census form in 2000. In addition, the Bureau changed the way 
coverage evaluation accounted for people who moved between Census Day 
and the day of the coverage evaluation interview. These design changes 
resulted in less complete information on people and likely contributed 
to an increase in imputations. (These and other changes are explained 
in more detail in app. II.) 

Because imputed records are simply added to the census totals and do 
not have names attached to them, it was impossible for A.C.E. to either 
count imputed individuals using the A.C.E. sample survey or incorporate 
them into the matching process. At any rate, since the true number and 
characteristics of these persons are unknown, matching these records 
via A.C.E. would not have provided meaningful information on coverage 
error. A.C.E. was designed to measure the net census coverage error, in 
essence the net effect of people counted more than once minus people 
missed, and included measurement of the effects of imputation on 
coverage error. The Bureau generalizes its estimates of coverage error 
to cover imputations and also maintains that its imputation methods do 
not introduce any statistical bias in population counts. But the 
formulas used by the Bureau to calculate its estimates of coverage 
error account for imputations by subtracting them from the census count 
being evaluated, not by measuring the error in them. And the Bureau did 
not attempt to determine the accuracy of the individual imputations, 
that is although the Bureau imputed persons they have reason to believe 
existed, the Bureau does not know whether it over-or underestimated 
such persons. As more imputations are included in the census total, the 
generalization of coverage error estimates to that total population 
becomes less reliable.

Similarly, the Bureau created an additional coverage error blind spot 
by including in the census 2.4 million records that it previously 
suspected were duplicates and thus were not included in the coverage 
evaluation. Prior to A.C.E. matching, the Bureau removed about 6 
million persons from the census population, identifying them as likely 
duplicates. Then, after completing additional research on these 
possible duplicates, the Bureau decided to reinstate the records for 
2.4 million of these persons it no longer suspected were duplicates. 
However, it did so after A.C.E. had completed matching and evaluating 
the census records from which the 2.4 million persons had been removed 
and for which coverage error estimation had begun (see fig. 3).

Figure 3: A.C.E. Matching Results Lacked Data on Suspected Duplicates 
That Were in the Census: 

[See PDF for image]

Note: Figures are not drawn to proportion.

[End of figure]

The Bureau documented in a memorandum that the exclusion of the records 
from A.C.E. processing was not statistically expected to affect A.C.E. 
results. However, later Bureau research concluded that over 1 million 
of these 2.4 million records were likely duplicates, none of which 
could have been detected by A.C.E. While the Bureau maintains that the 
reinstatement of the over 1 million reinstated likely duplicates did 
not affect the A.C.E. estimate in a statistically significant way, this 
suggests that the resulting A.C.E.-based estimate of national 
population itself is blind to the presence in the census of the over 1 
million duplicates the Bureau reintroduced. For 2010, Bureau officials 
have chartered a planning group responsible for, among other things, 
proposing improvements to reduce duplication in the census, which may 
address some of the problem.

In addition to excluding populations from the scope of evaluation, the 
Bureau further curtailed its ability to measure coverage error by 
reducing A.C.E.'s search area to only one geographic ring around 
selected A.C.E. sample areas during the matching process. For the 1990 
Census, the Bureau's coverage evaluation program always searched at 
least one surrounding ring and an even larger ring in rural areas. 
However, in 1999, before a panel of the National Academy of Sciences, 
Bureau officials announced and defended the decision to not expand the 
search area except in targeted instances, saying that Bureau research 
indicated that the additional matches found in 1990 from the expanded 
search areas did not justify the additional effort. In its comments on 
our draft report, the Department writes that more important than the 
size of the search area is maintaining "balance"--i.e., search areas 
must be used consistently both to identify people who have been missed 
and to identify people who have been counted in error (including 
duplicates). The Department also justified the decision to reduce the 
search area in 2000 from 1990 in part by stating, "in an expected value 
sense, the reduced search area would have affected" [emphasis added] 
the extra missed people and the extra miscounted people equally, or 
been balanced. However, later research discovered large numbers of the 
missed duplicates in the actual census by matching A.C.E. persons to 
census persons nationwide--far beyond the areas searched during A.C.E. 
matching. A 2001 Bureau report presenting the results of computer 
rematching of the A.C.E. sample concluded, "Our analysis found an 
additional 1.2 million duplicate enumerations in units that were out-
of-scope for the A.C.E. but would have been in-scope for [1990 coverage 
evaluation]."[Footnote 6] In other words, if the Bureau had continued 
its 1990 practice of searching in housing units in larger geographic 
areas in 2000, the A.C.E. process might have identified more duplicates 
and yielded better results.[Footnote 7] The Bureau research cited above 
appears to question the decision to reduce the search areas. In fact, 
after the 2000 Census was completed, again before the National Academy 
of Sciences, a Bureau official suggested that any coverage evaluation 
methods for 2010 should conduct a more thorough search, perhaps 
expanding the search area to two or more geographic rings everywhere.

Bureau Has Not Fully Accounted for How Design Decisions Affected 
Coverage Error Estimates: 

This review has identified only some of the decisions that could have 
created problems in A.C.E. estimates. Because the Bureau has not 
attempted to account for how all of its design decisions relating to 
A.C.E. and the census affected the outcome of the program, the full 
range of reasons that A.C.E. estimates were not reliable remains 
obscure.

Bureau research documented and this report describes the magnitude of 
the direct effects of most of these design decisions in terms of the 
size of the census population affected, and the Bureau's final 
reporting on the revised A.C.E. estimates mentions many design changes, 
but not together or comprehensively, and they do not explain how the 
changes might have affected the estimates of coverage error. Without 
clear documentation of how significant changes in the design of the 
census and A.C.E. might have affected the measurements of census 
accuracy, it is not apparent how problems that have arisen as a result 
of the Bureau's own decisions can be distinguished from problems that 
are less under the Bureau's control, i.e., difficulties inherent to 
conducting coverage evaluation. Thus the Bureau's plans to measure the 
coverage error for the 2010 Census are not based on a full 
understanding of the relationship between the separate decisions it 
makes about how to conduct A.C.E. and the census and the resulting 
performance of its coverage measurement program. This in turn could 
hamper the Bureau's efforts to craft a more successful coverage 
measurement program for the next national head count.

The Bureau Has Not Produced Reliable Revised Estimates of Coverage 
Error for the 2000 Census: 

While the Bureau produced a vast body of research regarding the census 
and A.C.E., including multiple reassessments and revisions of earlier 
work, the revised estimates are not reliable. The initial A.C.E. 
estimates of coverage error suggested that while historical patterns of 
differences in undercounts between demographic groups persisted, the 
Bureau had succeeded in 2000 in reducing the population undercounts of 
most minorities, and the subsequent revised estimates showed even 
greater success in reducing population undercounts in the Census. 
However, the large number of limitations described in the Bureau's 
documentation of the methodology used to generate the revised estimates 
of coverage error suggest that these estimates are less reliable than 
reported and may not describe the true rate of coverage error. The 
Bureau, however, has not made the full impact of these methodological 
limitations on the data clear. Moreover, the final revised estimates of 
coverage error for the count of housing units and the count of people, 
which the Bureau expected to be similar if the estimates were reliable, 
differed, further raising questions about the reliability of the 
revised estimates.

Bureau's Conclusion That A.C.E. Estimates Are Unreliable Is Based on 
Results That Are Unreliable Themselves: 

The Bureau undertook an extensive review of A.C.E.'s results over a 3-
year period. In doing so, the Bureau revised its estimates of A.C.E. 
coverage error twice--first in October 2001 and again in March 2003. 
These revisions suggest that the original A.C.E. estimates were 
unreliable. Figure 4 illustrates how each of the revised A.C.E. 
estimates of coverage error reduced the undercount for most of the 
three major race/origin groups from the initial A.C.E. estimates. Note 
that the final revised estimate indicates that the total population was 
actually overcounted by one-half of 1 percent.

Figure 4: Reported Undercount Estimates Generally Decreased with A.C.E. 
Revisions: 

[See PDF for image]

[End of figure]

The differences in the revised estimates presumably provide a measure 
of the error in the original A.C.E. estimates. (The estimated net 
population undercounts--and their standard errors--for these groups are 
provided in app. III.) However, the revised estimates of coverage error 
may not be reliable enough themselves to provide an adequate basis for 
such a comparison to measure error in the original A.C.E. estimates.

First, the number of Bureau-documented limitations with respect to the 
methodologies used in generating A.C.E.'s revised estimates raises 
questions about the accuracy of the revised estimates. Within 
voluminous technical documentation of its process, the Bureau 
identified several methodological decisions wherein if the decisions 
had been made differently, they may have led to appreciably different 
results. Thus the methods the Bureau chose may have affected the 
estimates of census coverage error themselves and/or the measures of 
uncertainty associated with the estimates, limiting the general 
reliability of the revisions. The limitations in the methodologies for 
the revised estimates included the following: 

* Errors in the demographic data used to revise estimates may have 
contributed to additional error in the estimates.

* Errors stemming from the Bureau's choice of model to resolve 
uncertain match cases were accounted for in initial March 2001 A.C.E. 
estimates but were not accounted for in the revised estimates in March 
2003.

* Alternative possible adjustments for known inefficiencies in computer 
matching algorithms would directly affect revised estimates.

* The Bureau's evaluations of the quality of clerical matching were 
used to revise the initial A.C.E. coverage estimates, leaving the 
Bureau less reliable means to measure the uncertainty in the revised 
estimates.

The Bureau Has Not Clearly Quantified and Reported the Full Impact of 
the Methodological Limitations on the Revised Estimates: 

For the earlier revision of coverage error estimates, the Bureau 
provided the range of impacts that could result from some different 
methodological decisions, enabling more informed judgments regarding 
the reliability of the data. For example, in support of its October 
2001 decision to not adjust census data, the Bureau showed that 
different assumptions about how to treat coverage evaluation cases that 
the Bureau could not resolve could result in variations in the census 
count of about 6 million people. The Bureau also had reported 
previously the range of impacts on the estimates resulting from 
different assumptions and associated calculations to account for the 
inefficiency of computer matching. They found that different 
assumptions could result in estimates of census error differing by 
about 3.5 million people.

However, with the final revision of the A.C.E. coverage error 
estimates, the Bureau did not clearly provide the ranges of impact 
resulting from different methodological decisions. While the Bureau did 
discuss major limitations and indicated their uncertain impact on the 
revised estimates of coverage error, the Bureau's primary document for 
reporting the latest estimates of coverage error did not report the 
possible quantitative impacts of all these limitations--either 
separately or together--on the estimates. Thus readers of the reported 
estimates do not have the information needed to accurately judge the 
overall reliability of the estimates, namely, the extent of the 
possible ranges of the estimates had different methodological decisions 
been made.

Sampling errors were reported alongside estimates of census error, but 
these do not adequately convey the extent of uncertainty associated 
with either the reported quantitative estimates themselves or the 
conclusions to be drawn from them. For example, the Bureau decided to 
make no adjustment to account for the limitation of computer matching 
efficiency when calculating its latest revision of estimates of 
coverage error, unlike the adjustment it made when calculating its 
earlier revised estimates. When justifying its adjustment made in its 
earlier revised estimates, the Bureau demonstrated that the choice of 
adjustment mattered to the calculated results. But the potential 
significance to the calculated results of the Bureau's having made a 
different assumption was not reflected in the Bureau's primary 
presentation of its estimates and their errors. The absence of clear 
documentation on the possible significant impacts of such assumptions 
could lead readers of the Bureau's reporting to believe erroneously 
that all assumptions have been accounted for in the published 
statistics, or that the estimates of coverage error are more reliable 
than they are.

Unexpected Differences in Patterns of Coverage Error between the 
Housing and the Population Count Further Call into Question Reliability 
of Revised Estimates: 

According to Bureau reporting, when it examined the validity of the 
revised coverage error estimates the Bureau expected to see across 
demographic groups similar patterns between the coverage error for the 
count of the population and the count of housing. That is, if a 
population was overcounted or undercounted, then the count of housing 
units for that population was expected to be overcounted or 
undercounted as well. The Bureau developed estimates of coverage error 
in the count of housing units from A.C.E. data. But the comparisons of 
non-hispanic blacks and hispanics to non-hispanic whites in figure 5 
shows that the relative housing undercounts are opposite of what was 
expected by the Bureau.

Figure 5: Population and Housing Undercounts: 

[See PDF for image]

[End of figure]

For example, the estimated population undercount for non-hispanic 
blacks is almost 3 percent greater than that of the majority group--
non-hispanic white or other--but the estimated housing unit undercount 
for non-hispanic blacks is about 0.8 percent less than that of the 
majority group. In addition, while the Bureau estimated that the non-
hispanic white majority group had a net population overcount of over 1 
percent, the Bureau estimated the majority group as having its housing 
units undercounted by about one-third of a percent. (The estimated net 
housing unit undercounts--and their standard errors--for these groups 
are provided in app. III.) 

Bureau officials told us that the problems A.C.E. and the census 
experienced with identifying coverage error in the population do not 
seem likely to have affected housing counts. However, when estimating 
coverage error for housing units for specific populations (e.g., by 
gender or race/ethnicity) errors in the population count can affect the 
reliability of housing tabulations. This is because when the Bureau 
tabulates housing data by characteristics like gender or race, it uses 
the personal characteristics of the person recorded as the head of the 
households living in each housing unit. So if there are problems with 
the Bureau's count of population for demographic groups, for example by 
gender or sex, they will affect the count of housing units for 
demographic groups. While the unexpected patterns in population and 
housing unit coverage error may be reconcilable, Bureau officials do 
admit that problems with the estimations of population coverage error 
may also adversely affect the reliability of other measures of housing 
count accuracy they rely upon, such as vacancy rates. Bureau officials 
have indicated the need to review this carefully for 2010.

Bureau Reports Revised Estimates Resulted in Lessons Learned: 

While the multiple reassessments and revisions of earlier work did not 
result in reliable estimates, these efforts were not without value, 
according to the Bureau. Bureau officials stated that the revision 
process and results helped the Bureau focus for 2010 on detecting 
duplicates, revising residence rules, and improving the quality of 
enumeration data collected from sources outside the household, such as 
neighbors, as well as providing invaluable insights for its program of 
updating census population estimates throughout the decade.

Conclusions: 

The volume and accessibility over the Internet of the Bureau's research 
may have made this the most transparent coverage evaluation exercise of 
a Decennial Census. However, as the Bureau has closed the book on 
Census 2000 and turned toward 2010, the reliability of the Bureau's 
coverage estimates remains unknown. The Bureau made extensive efforts 
to evaluate the census and its coverage error estimates resulting from 
A.C.E., but these efforts have not been sufficient to provide reliable 
revised estimates of coverage error. So while much is known about 
operational performance of the 2000 Census, one of the key performance 
measures for the 2000 census remains unknown.

Moreover, neither Congress nor the public know why the coverage 
evaluation program did not work as intended, because the Bureau has not 
provided a clear accounting of how census and A.C.E. design decisions 
and/or limitations in the A.C.E. revision methodology discussed in this 
report accounted for the apparent weakness--or strengths--of A.C.E. 
Without such an accounting, the causes of problems and whether they can 
be addressed will remain obscure. And as the Bureau makes plans for 
coverage evaluation for the 2010 Census, whether that program 
approximates A.C.E.'s design or not, the Bureau will be missing 
valuable data that could help officials make better decisions about how 
to improve coverage evaluation.

Finally, this lack of information calls into question the Bureau's 
claim (made in response to a prior GAO recommendation that the Bureau 
determine the feasibility of adjustment) that it has already 
established that using coverage evaluation for adjustment purposes is 
not feasible. Without clearly demonstrating what went wrong with its 
most recent coverage evaluation and why, the Bureau has not shown that 
coverage evaluation for the purpose of adjustment is not feasible. In 
fact, this report mentions two census improvements--to residence rules 
and to efforts to identify and reduce duplicates--that the Bureau is 
already considering that could make A.C.E. estimates more reliable, and 
perhaps even feasible. Furthermore, although the Bureau reports that 
its experience with revising A.C.E. estimates has provided lessons, it 
remains unclear how the Bureau will use its published coverage error 
estimates to make decisions leading to a more reliable measure of 
coverage error in 2010, or how the unreliable estimates can be of value 
to policymakers or the public.

Recommendations for Executive Action: 

As the Bureau plans for its coverage evaluation of the next national 
head count in 2010, we recommend that the Secretary of Commerce direct 
that the Bureau take the following three actions to ensure that 
coverage evaluation results the Bureau disseminates are as useful as 
possible to Congress and other census stakeholders: 

* To avoid creating any unnecessary blind spots in the 2010 coverage 
evaluation, as the Bureau plans for its coverage evaluation in 2010, it 
should take into account how any significant future design decisions 
relating to census (for example, residence rules, efforts to detect and 
reduce duplicates, or other procedures) or A.C.E. (for example, scope 
of coverage, and changes in search areas, if applicable), or their 
interactions, could affect the accuracy of the program.

* Furthermore, in the future, the Bureau should clearly report in its 
evaluation of A.C.E. how any significant changes in the design of 
census and/or A.C.E. might have affected the accuracy of the coverage 
error estimates.

* In addition GAO recommends that in the future the Bureau plan to not 
only identify but report, where feasible, the potential range of impact 
of any significant methodological limitation on published census 
coverage error estimates. When the impact on accuracy is not readily 
quantifiable, the Bureau should include clear statements disclosing how 
it could potentially affect how people interpret the accuracy of the 
census or A.C.E.

Agency Comments and Our Evaluation: 

The Under Secretary for Economic Affairs at the Department of Commerce 
provided us written comments from the Department on a draft of this 
report on September 10, 2004 (see appendix I). The Department concurred 
with our recommendations, but took exception to some of our analyses 
and conclusions and provided additional related context and technical 
information. In several places, we have revised the final report to 
reflect the additional information and provided further clarification 
on our analyses.

The Department was concerned that our draft report implied that A.C.E. 
was inaccurate because it should have measured gross coverage error 
components, and that this was misleading because the Bureau designed 
A.C.E. to measure net coverage errors. While we have previously 
testified that the Bureau should measure gross error components, and 
the Department in its response states that this is now a Bureau goal 
for 2010, we clarified our report to reflect the fact that the Bureau 
designed A.C.E. to measure net coverage error.

Further, although the Department agreed with our finding that the 
Bureau used residence rules that were unable to capture the complexity 
of American society thus creating difficulty for coverage evaluation, 
the Department disagreed with our characterization of the role four 
other census and A.C.E. design decisions played in affecting coverage 
evaluation. Specifically, the Bureau does not believe that any of the 
following four design decisions contributed significantly to the 
inaccuracy of the A.C.E. results: 

1. The Treatment of the Group Quarters Population--The Department 
commented that we correctly noted that group quarter residents were 
excluded from the A.C.E. universe who would have been within the scope 
of A.C.E. under the 1990 coverage evaluation design, and that a large 
number of these people were counted more than once in 2000. The 
Department maintains that the Bureau designed A.C.E. to measure such 
duplicates. We believe this is misleading. As the Department noted, 
during its follow-up at housing units the Bureau included questions 
intended to identify the possible duplication of college students 
living away at college, and we have now included this in our final 
report. But as we stated in our draft report, A.C.E. did not provide 
coverage error information for the national group quarters population. 
Moreover, during A.C.E. the Bureau did not search for duplicate people 
within the group quarters population counted by the census, as it did 
within housing units counted by the census. In fact, later Bureau 
research estimated that if it had done so, the Bureau would have 
identified over 600,000 additional duplicates there. As such, our 
finding that this may have contributed to the unreliability of coverage 
error estimates still stands.

2. The Treatment of Census Imputations--The Department stated that 
A.C.E. was designed to include the effects of imputations on its 
measurement of coverage error and that there was no basis for our draft 
report stating that as more imputations were included in the census 
then coverage error estimates became less reliable. While we agree that 
the Bureau's estimates of coverage error accounted for the number of 
imputations, as we report, and as the Department's response reiterated, 
no attempt was made to determine the accuracy of the imputations 
included in the census. Thus any errors in either the number or 
demographic characteristics of the population imputed by the Bureau 
were not known within the coverage error processes or estimation. As a 
result, in generalizing the coverage error estimates to the imputed 
segment of the population, the Bureau assumed that the imputed 
population had coverage error identical to the population for which 
coverage error was actually measured. Furthermore, the larger the 
imputed segment of the population became the more this assumption had 
to be relied upon. Since the real people underlying any imputations are 
not observed by the census, the assumption is, in its strictest sense, 
untestable, thus we maintain that increasing the number of imputations 
included in the census may have made generalizing the coverage error 
estimates to the total census population less reliable.

3. The Treatment of Duplicate Enumerations in the Reinstated Housing 
Units--The Department wrote that our draft report incorrectly 
characterized the effects of reinstating duplicates into the census. 
The Department indicated that A.C.E., having been designed to measure 
net coverage error, treated the over 1 million likely duplicates 
"exactly correctly" and that including them in the census had no effect 
on the mathematical estimates of coverage error produced by A.C.E. We 
reported that, according to Bureau research, introducing the additional 
duplicates into the census appeared to have no impact on the A.C.E. 
estimates. But we view this fact as evidence of a limitation, or blind 
spot, in the Bureau's coverage evaluation. The fact that 2.4 million 
records, containing disproportionately over 1 million duplicate people 
could be added to the census without affecting the A.C.E. estimates 
demonstrates a practical limitation of those coverage error estimates. 
We maintain that the resultant measure of coverage error cannot be 
reliably generalized to the entire population count of which those 
1 million duplicates are a part.

4. Size of the Search Area--The Department wrote that a search area 
like that used in 1990 would have done little to better measure the 
number of students and people with vacation homes who may have been 
duplicated in 2000. It described our conclusion regarding the reduction 
in search area from 1990 as not supported by the relative magnitudes of 
these situations. And finally, the Department offered additional 
support for the decision to reduce the search area by describing the 
reduced search area as balanced, or "in an expected value sense" 
[emphasis added] affecting the number of extra missed people and the 
extra miscounted people equally.

In our final report we added a statement about the Department's concern 
over the importance of balance in its use of search areas. But we 
disagree that our conclusion is unsupported, since in our draft report 
we explicitly cited Bureau research that found an additional 
1.2 million duplicate enumerations in units that were out-of-scope for 
2000 A.C.E. but that would have been in-scope for 1990's coverage 
evaluation.

In addition, the Department offered several other comments.

Regarding our finding that the Bureau has not produced reliable revised 
estimates of coverage error for the 2000 Census, and, specifically, 
that the full impact of the Bureau's methodological limitations on the 
revised estimates has not been made clear, the Department wrote that 
the Census Bureau feels that further evaluations would not be a wise 
use of resources. We concur, which is why our recommendations look 
forward to the Bureau's preparation for 2010.

The Department commented that it did not see how we could draw 
conclusions about the reliability of the Bureau's coverage evaluation 
estimates if we did not audit the underlying research, data, or 
conclusions. We maintain that the objectives and scope of our review 
did not require such an audit. As we described, and at times cited, 
throughout our draft report, we used the results of the Bureau's own 
assessment of the 2000 Census and its coverage evaluation. That 
information was sufficient to draw conclusions about the reliability of 
the A.C.E. estimates. As a result, there was no need to verify 
individual Bureau evaluations and methodologies.

The Department expressed concern that our draft report implied that the 
unexpected differences in patterns of coverage error between the 
housing and the population count were irreconcilable. That was not our 
intent, and we have clarified that in the report.

The Department expressed concern over the report's characterization of 
the 1990 coverage error estimates for group quarters as weak in part 
due to the high mobility of this population. However, the 1990 group 
quarters estimates are described as "weak" in a Bureau memorandum 
proposing that group quarters be excluded from the 2000 coverage 
evaluation. The memorandum also explains how the mobility within the 
group quarters population contributes to the resulting estimates. We 
have not revised the characterization of the group quarters coverage 
error estimates or the causal link due to the mobility of that 
population, but we have revised our text to state more clearly that the 
1990 estimates being discussed are those for group quarters.

As agreed with your offices, unless you release its contents earlier, 
we plan no further distribution of this report until 30 days from its 
issue date. At that time we will send copies to other interested 
congressional committees, the Secretary of Commerce, and the Director 
of the U.S. Census Bureau. Copies will be made available to others upon 
request. This report will also be available at no charge on GAO's Web 
site at [Hyperlink, http://www.gao.gov].

If you or your staff have any questions concerning this report, please 
contact me on (202) 512-6806 or by e-mail at daltonp@gao.gov or 
Robert Goldenkoff, Assistant Director, at (202) 512-2757 or 
goldenkoffr@gao.gov. Key contributors to this report were Ty Mitchell, 
Amy Rosewarne, and Elan Rohde.

Signed by: 

Patricia A. Dalton: 
Director, Strategic Issues: 

[End of section]

Appendixes: 

Appendix I: Comments from the Department of Commerce: 

UNITED STATES DEPARTMENT OF COMMERCE: 
The Under Secretary for Economic Affairs: 
Washington, D.C. 20230:

SEP 10 2004:

Ms. Patricia A. Dalton: 
Director, Strategic Issues: 
U.S. Government Accountability Office: 
Washington, DC 20548:

Dear Ms. Dalton:

The U.S. Department of Commerce appreciates the opportunity to comment 
on the U.S. Government Accountability Office draft report entitled 2000 
Census: Design Choices Contributed to Inaccuracy of Coverage Evaluation 
Estimates (GAO-04-468). The Department's comments on this report are 
enclosed.

Sincerely,

Signed by: 

Kathleen B. Cooper:

Enclosure:

Comments from the U.S. Department of Commerce Regarding the U.S. 
Government Accountability Office (GAO) Draft Report Entitled 2000 
Census: Design Choices Contributed to Inaccuracy of Coverage Evaluation 
Estimates (GAO-04-468):

Thank you for providing us the opportunity to comment on your draft 
report 2000 Census: Design Choices Contributed to Inaccuracy of 
Coverage Evaluation Estimates (GAO-04-468).

Response to GAO Recommendations:

We concur with the GAO's recommendations regarding planning for the 
2010 census. However, we have several comments on the analyses and 
conclusions in this report.

Major Comments on GAO's Analyses and Conclusions:

First, the Accuracy and Coverage Evaluation Program (A.C.E.) was 
designed explicitly and very publicly to measure net census coverage 
error. The GAO report implies that the A.C.E. should have had different 
goals-specifically, that the A.C.E. should have been designed to 
measure gross coverage error components, including providing a direct 
measure of census duplicates. While the U.S. Census Bureau now has made 
this a goal for 2010, it is misleading to imply that the 2000 census 
A.C.E. was inaccurate because it chose to measure net rather than gross 
coverage errors.

Second, as the Census Bureau has stated and documented elsewhere, the 
estimates produced by the A.C.E. in March 2001 were not an accurate 
measure of census coverage. This inaccuracy occurred in spite of the 
outstanding execution of the design by the Census Bureau's field staff, 
clerical processing staff, and information technology staff. Therefore, 
we agree with the GAO's primary conclusion that the inaccuracy can only 
be attributed to the assumptions and decisions that underlay the 
design.

However, it is important for future coverage measurement programs that 
the true design failures be understood. The GAO report points to five 
design decisions as critical to the lack of success of the A.C.E. The 
Census Bureau agrees that we had difficulties measuring residency 
status, but we cannot agree with the GAO's analysis of the other four 
design decisions nor with its conclusion. In fact, the Census Bureau 
views this report as misleading in its descriptions of the nature and 
source of the A.C.E. errors and believes that the GAO's conclusions are 
based on Census Bureau research results that have been taken out of 
context.

The design of the A.C.E. was based in part on a key assumption-that 
through field interviewing, each person could be assigned to a correct 
(or at least consistently to a unique) location at which the person 
should have been enumerated. In the case of people enumerated at two 
different locations, the design relied in most cases upon information 
provided by respondents to determine which location should be 
considered the correct Census Day address and which should be 
considered an erroneous enumeration. It is important to note that, 
because of the statistical nature of the survey, each determination was 
made separately. This design, and that underlying assumption, seemed to 
have worked reasonably well in previous censuses. However, based on 
subsequent analysis, it is now clear that for the 2000 A.C.E., this 
assumption frequently did not hold. As a result, we found a number of 
cases where, for a person enumerated at two different locations, our 
separate determinations of census day address both concluded that the 
person was correctly enumerated and thus did not detect an erroneous 
enumeration. Please see the Executive Steering Committee for A.C.E. 
Policy II (ESCAP) Report 6 entitled "ESCAP II: Census Person 
Duplication and the Corresponding A.C.E. Enumeration Status." This 
report can be found at http://www.census.gov/dmd/www/pdf/Rgport6.pdf 
The Census Bureau remains convinced that the failure of this assumption 
(that we could gather accurate information about Census Day residence) 
was the principal cause of A.C.E. inaccuracy in 2000.

The GAO report, however, lists four other design decisions (which they 
refer to as "blind spots") as the cause of A.C.E. inaccuracy: (1) the 
treatment of the group quarters population, (2) the treatment of census 
imputations, (3) the treatment of duplicate enumerations in the 
reinstated housing units, and (4) size of the search area. The Census 
Bureau, however, does not believe that any of these decisions 
contributed significantly to the inaccuracy of the A.C.E.

(1) The Treatment of the Group Quarters Population-The GAO report 
correctly notes that non-institutional group quarters residents (e.g., 
students in college dorms) were included in the 1990 post-enumeration 
survey (PES) universe, but excluded from the 2000 A.C.E. universe. It 
is also true that analysis done after 2000 indicates a large number of 
people duplicated between the housing unit and the group quarters 
population-especially college students. However, what the report fails 
to make clear is that the A.C.E. was designed to measure such 
duplicates. Specifically, the design called for careful questioning of 
the student or his/her family to determine whether he/she was away at 
college. Evidence now indicates that this questioning failed to reveal 
that many students were counted at home, but should have been counted 
at college. There was never an intention to directly match between the 
home address and all possible college residences. This was certainly 
not done in 1990. In any case, including group quarters in the universe 
for A.C.E. would have had no effect on how persons responded to the 
A.C.E. questioning described above. Our collection and treatment of 
those responses-not the universe definition-are what led to the errors 
in the A.C.E. estimates.

(2) The Treatment of Census Imputations-The Census Bureau agrees with 
the finding that there were more whole-person records imputed in Census 
2000 than in the 1990 census, and that this is something that the 
Census Bureau must research. However, the A.C.E. was properly designed 
to account for this effect. The A.C.E. was explicitly designed to 
measure net census coverage error, including the effects on net error 
of the census imputations. There was no attempt to determine the 
accuracy of individual census imputations. There is no basis for GAO's 
claim that "as more imputations are included in the census total, the 
generalized coverage error estimates become less reliable."

(3) The Treatment of Delicate Enumerations in the Reinstated Housing 
Units-The GAO report is particularly incorrect in its characterization 
of the effect of any duplicate enumeration reinstated to the census 
files after A.C.E. processing. Specifically, the following statement by 
the GAO cannot be sustained: "While the Bureau maintains that the 
reinstatement of the over 1 million reinstated likely duplicates did 
not affect the A.C.E. estimate in a statistically significant way, this 
suggests that the A.C.E. estimate itself is blind to the presence of 
the over 1 million duplicates the Bureau introduced."

Again, the A.C.E. was designed explicitly to measure the net coverage 
error. Although not specifically identified as duplicates, these "over 
1 million" cases were treated exactly correctly. Indeed, setting these 
duplicate cases aside for the purpose of dual system estimation of net 
error is exactly how they would and should have been treated even if 
they had been separately identified. The mathematical treatment is 
identical, with the exception that by setting aside these 1 million 
cases directly, sampling error actually was reduced.

(4) Size of the Search Area-As noted by the GAO, the effective search 
area (for matching Census and A.C.E. interviews) was reduced from what 
it had been in 1990. However, this would have affected both the number 
of census omissions we measured and the number of census erroneous 
enumerations (including duplicates) we detected. Because, in an 
expected value sense, the reduced search area would have affected these 
components equally, this was a reasonable decision given our goal of 
measuring the net coverage error. Expanding the search area certainly 
would have resulted in a few more duplicate enumerations being detected 
and measured, but also would have resulted in a few less omissions 
being measured. In any case, the additional duplicates identified by 
expanding the search area only would be those situations where both 
enumerations were geographically near each other. An effective search 
area like that used in 1990 would have done little to better measure 
the number of students counted both at home and at college, or people 
counted both at their usual home and at a vacation home. The GAO's 
conclusion (that the size of the search area was a leading cause of 
A.C.E. inaccuracy in 2000) is not supported by the relative magnitudes 
of these various situations.

Furthermore, the size of the search area is not the relevant issue. 
Rather, what matters is whether there was a consistent treatment of the 
search area used to identify missed people and erroneous enumerations. 
Any inconsistencies could result in balancing error, which was 
identified as a potential problem in the first Executive Steering 
Committee For A.C.E. Policy (ESCAP) report and subsequently was 
extensively evaluated. As noted in the second ESCAP report, these 
evaluations indicated that the dual-system estimates may have 
overstated the net undercount by 450,000 persons and that this was 
mainly a result of geocoding errors.

In summary, while other-and equally reasonable-decisions could have 
been made with respect to these four "blind spots," none of these 
design choices we made for 2000 were the primary cause of A.C.E. 
inaccuracy.

Other Comments:

The GAO report states that with respect to the A.C.E. Revision II 
estimates, the Census Bureau "has not made the full impact of these 
methodological limitations on the data clear" [Page 16]. In response to 
this conclusion, the Census Bureau believes it is important to review 
the history of our efforts. The Census Bureau conducted Census 2000 and 
evaluated it through the A.C.E. The Census Bureau then conducted an 
evaluation of the A.C.E. When this evaluation raised questions, it 
itself was evaluated. Then, taking into account all previous 
evaluations, the Revision II estimates were prepared and their 
limitations carefully, thoroughly, and publicly documented. Although 
the Census Bureau may not have explicitly quantified any remaining non-
sampling errors, the A.C.E. evaluations, research, and revision efforts 
have provided valuable information about erroneous census enumerations 
and have identified major issues with data collection for both the 
census and the coverage measurement survey. The Census Bureau feels 
that further evaluations would not be a wise use of resources.

Although the GAO notes "we did not audit the Bureau's research, the 
underlying data or its conclusions," the report concludes that the 
A.C.E. Revision II estimates "are not reliable." We do not see how this 
conclusion can be supported, particularly if the GAO did not audit our 
research or data.

The report raises some interesting questions concerning the differences 
between the person coverage estimates and the housing unit coverage 
estimates. These differences warrant further careful study. However, 
the report implies that these differences are, prima facie, 
unreconcilable, because "the Bureau expected to see across demographic 
groups similar patterns between the coverage error for the count of the 
population and the count of housing units." This ignores within-
household error. Net undercount for the Black population can be fully 
consistent with net overcount for the Black housing unit count, 
possibly due to high within-household omissions. Net overcount of the 
non-Hispanic white and other population can be fully consistent with an 
undercount of the housing units, due to, for example, large duplication 
of college students.

The report states that "the Bureau reported that the high mobility of 
these [Group Quarters] population made it more difficult to count them, 
thus the 1990 estimates of coverage were weak" [page 12]. It is true 
that the high mobility of this population made 1990 matching more 
difficult, and that there is more uncertainty in the coverage estimates 
for this segment of the population. It does not follow that having 
included a highly mobile and difficult-to-count group significantly 
weakened the overall 1990 estimates. The move to exclude this group in 
2000 grew mainly from a decision made on other grounds to treat movers 
differently in 2000 than was done in 1990, although there were concerns 
about the difficulties associated with this group. 

[End of section]

Appendix II: Bureau Design Decisions Increased Imputations: 

The Bureau made various design decisions that resulted in an increase 
in the number of "imputations"--or people guessed to exist--included in 
the census population that could not be included within the A.C.E. 
sample survey. The Bureau believes certain numbers of people exist 
despite the fact that the census records no personal information on 
them; thus it projects, via computer-executed algorithms, numbers and 
characteristics of people and includes them in the census. Such records 
are simply added to the census totals, and do not have names attached 
to them. Thus it was impossible for A.C.E. to either count imputed 
individuals using the A.C.E. sample survey or incorporate them into the 
matching process. Since the true number and the characteristics of 
these persons are unknown, matching nameless records via A.C.E. would 
not have provided any meaningful information on coverage evaluation. 
The number of people the Bureau imputed grew rapidly in 2000, from 
about 2 million in 1990 to about 5.

One of the reasons for the large increase in imputations may be the 
decision by the Bureau to eliminate field edits--the last-minute 
follow-up operation to collect additional information from mail-back 
forms that had too little information on them to continue processing--
from field follow-up in 2000. While acknowledging that this decision 
may have increased imputations for 2000, a senior Bureau official 
justified the decision by describing the field edits in 1990 as 
providing at times a "clerical imputation" that introduced a subjective 
source of error, which computer-based imputation in 2000 lacked.

The Bureau also reduced the number of household members for whom 
personal information could be provided on standard census forms, and 
this also contributed to the increase in imputations. Households 
reporting a household size greater than 6 in 2000--the number for whom 
personal characteristics could be provided--were to be automatically 
contacted by the Bureau to collect the additional information. Yet not 
all large households could be reached for the additional information, 
and the personal characteristics of the remaining household members 
needed to be imputed. Again, A.C.E. would have been unable to match 
people counted by its sample survey to imputations, so imputed people 
were excluded from A.C.E. calculations of coverage errors.

An A.C.E. design choice by the Bureau that likely increased the amount 
of data imputed within the A.C.E. sample survey was how the Bureau 
decided to account for people who moved between Census Day and the day 
of the A.C.E. interview. Departing from how movers were dealt with in 
1990, and partly to accommodate the initial design for the 2000 Census, 
which relied on sampling nonrespondents to the census, for 2000 the 
Bureau relied on the counts of the people moving into A.C.E. sample 
areas to estimate the number of matched people who had actually lived 
in the A.C.E. areas on Census Day but moved out. This decision resulted 
in the Bureau having less complete information about the Census Day 
residents in A.C.E. sample areas who had moved out, and likely 
increased the number of imputations that were later required, making it 
more difficult to match these moving persons to the census. A Bureau 
official also cited this decision as partial justification for not 
including group quarters in A.C.E. search areas.

The extent that imputation affected the accuracy of the census is 
unknown. The National Academy of Sciences discussed in an interim 
report on the 2000 Census the possibility of a subset of about 1.2 
million of these imputations being duplicates.[Footnote 8] That report 
stated that, for example, "it is possible that some of these cases--
perhaps a large proportion--were erroneous or duplicates," and 
described another subset of about 2.3 million that could include 
duplicates. However, this Academy report did not include any data to 
suggest the extent of duplicates within these groups, and it may 
similarly have been possible for the number of persons in this group to 
have been underestimated. The Bureau maintains that the imputations 
were necessary to account for the people its field operations led it to 
believe had been missed, and that its imputation methods do not 
introduce statistical bias.

[End of section]

Appendix III: Bureau Estimates of Population and Housing Unit Net 
Undercounts: 

As shown in Table 1, the initial A.C.E. results suggested that the 
differential population undercounts of non-Hispanic blacks and 
Hispanics--the difference between their undercount estimate and that of 
the majority groups--persisted from Bureau estimates from its coverage 
evaluation in 1990. Yet they also demonstrated that the Bureau had 
apparently succeeded in reducing the magnitude of those differences 
since its evaluation of the 1990 Census.[Footnote 9]

Table 1: Estimated Percent Population Undercounts for Selected Race/
Origin Groups: 

Non-Hispanic black; 
Estimate in % (standard error in %): 1990 Estimates[A]: 4.57 (0.55); 
Estimate in % (standard error in %): A.C.E. (March 2001): 2.17 (0.35); 
Estimate in % (standard error in %): Revision I (October 2001): 0.78 
(0.45); 
Estimate in % (standard error in %): Revision II (March 2003): 1.84 
(0.43).

Hispanic; 
Estimate in % (standard error in %): 1990 Estimates[A]: 4.99 (0.82); 
Estimate in % (standard error in %): A.C.E. (March 2001): 2.85 (0.38); 
Estimate in % (standard error in %): Revision I (October 2001): 1.25 
(0.54); 
Estimate in % (standard error in %): Revision II (March 2003): 0.71 
(0.44).

Other; 
Estimate in % (standard error in %): 1990 Estimates[A]: --; 
Estimate in % (standard error in %): A.C.E. (March 2001): 0.73 (0.14); 
Estimate in % (standard error in %): Revision I (October 2001): -0.28 
(0.20); 
Estimate in % (standard error in %): Revision II (March 2003): --.

Non-Hispanic white; 
Estimate in % (standard error in %): 1990 Estimates[A]: 0.68 (0.22); 
Estimate in % (standard error in %): A.C.E. (March 2001): --; 
Estimate in % (standard error in %): Revision I (October 2001): --; 
Estimate in % (standard error in %): Revision II (March 2003): -1.13 
(0.20).

Total; 
Estimate in % (standard error in %): 1990 Estimates[A]: 1.61 (0.20); 
Estimate in % (standard error in %): A.C.E. (March 2001): 1.18 (0.13); 
Estimate in % (standard error in %): Revision I (October 2001): 0.06 
(0.18); 
Estimate in % (standard error in %): Revision II (March 2003): -0.49 
(0.20).

Sources: A.C.E. and Revision I data are reported in Basis of "Revised 
Early Approximation" of Undercounts Released Oct. 17, 2001 (Oct. 26, 
2001). 1990 and Revision II data are from PP-54 A.C.E. Revision II: 
Summary of Estimated Net Coverage (Dec. 31, 2002), Table 1.

Notes: The reported net undercounts for 2000 Census are for household 
population only. The reported 1990 net undercount also covers the 
population living in noninstitutional, nonmilitary group quarters.

[A] Tabulations by race/origin differed in 1990 from those for the 2000 
Census. The estimates reported in this column were labeled in 1990 as 
"Black", "Hispanic", and "Non-Hispanic White & Other", which also 
included "American Indian off Reservation".

[End of table]

Subsequent revised results published in October 2001 for three race/
origin groups indicated that differential undercounts were generally 
lower than the initial A.C.E. estimates, but that only the undercount 
estimate for Hispanics was still statistically different from zero. 
Finally, the latest revised estimates of undercount reported in March 
2003 that of these three major race/origin groups, only the non-
Hispanic black and non-Hispanic white percentage undercounts were 
significantly different from zero, in addition to the national net 
overcount.

Unlike the estimates of census population accuracy, which were revised 
twice since initial estimates, the census housing count accuracy 
estimates have not been revised and are based on the initial A.C.E. 
data. A subset of those results, including those provided here, were 
also published in October 2001.

Table 2: Estimated Percent Occupied Housing Undercounts Differ from 
Population Undercounts for Selected Race/Origin Groups: 

Non-Hispanic black; 
Estimate in % (standard error in %): Housing undercount: -0.45 
(0.29); 
Estimate in % (standard error in %): Population undercount: 1.84 
(0.43).

Hispanic; 
Estimate in % (standard error in %): Housing undercount: 0.06 
(0.35); 
Estimate in % (standard error in %): Population undercount: 0.71 
(0.44).

Non-Hispanic white or "some other race"; 
Estimate in % (standard error in %): Housing undercount: 0.38 
(0.14); 
Estimate in % (standard error in %): Population undercount: -1.13 
(0.20).

Total; 
Estimate in % (standard error in %): Housing undercount: 0.27 (0.13); 
Estimate in % (standard error in %): Population undercount: - 0.49 
(0.20). 

Source: DSSD A.C.E. REVISION II MEMORANDUM SERIES #PP-50 Comparison of 
A.C.E. Revision II Population Coverage Results with HUCS Housing 
Coverage Results J. Gregory Robinson Population Division Glenn S. 
Wolfgang Decennial Statistical Studies Division (December 31, 2002).

Notes: The reported net housing undercounts are for housing units only 
and do not include group quarters.

Net undercounts reported here are referred to by the Bureau as "single 
cell dual system estimates".

[End of table]

[End of section]

Glossary: 

This glossary is provided for reader convenience, not to provide 
authoritative or complete definitions.

Accuracy and Coverage Evaluation: 

The Bureau's Accuracy and Coverage Evaluation (A.C.E.) program was 
intended to measure coverage error (see below) for the 2000 Decennial 
Census. The program was to enable the Bureau to more accurately 
estimate the rate of coverage error via a sample survey of select areas 
nationwide, and if warranted, to use the results of this survey to 
adjust census estimates of the population for nonapportionment 
purposes.

Census Adjustment: 

The use of statistical information to adjust official census data.

Count of Housing: 

A tally of certain kinds of dwellings, including single-family homes, 
apartments, and mobile homes, along with demographic information on the 
inhabitants.

Count of Population: 

The headcount of everybody in the nation, regardless of their dwelling.

Coverage Error: 

The extent that minority groups are over-or undercounted in comparison 
to other groups in the census.

Coverage Evaluation: 

Statistical studies to evaluate the level and sources of coverage error 
in censuses and surveys.

Duplicates: 

When the census erroneously counts a person more than once.

Residence Rules: 

The rules the Bureau uses to determine where people should be counted.

[End of section]

Related GAO Products: 

2010 Census: Cost and Design Issues Need to Be Addressed Soon. 
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-04-37] 
(Washington, D.C.: January 15, 2004).

2000 Census: Coverage Measurement Programs' Results, Costs, and Lessons 
Learned. 
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-03-287] 
(Washington, D.C.: January 29, 2003).

2000 Census: Complete Costs of Coverage Evaluation Programs Are Not 
Available. 
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-03-41] 
(Washington, D.C.: October 31, 2002).

2000 Census: Coverage Evaluation Matching Implemented as Planned, but 
Census Bureau Should Evaluate Lessons Learned. 
[Hyperlink, http:// www.gao.gov/cgi-bin/getrpt?GAO-02-297] 
(Washington, D.C.: March 14, 2002).

2000 Census: Coverage Evaluation Interviewing Overcame Challenges, but
Further Research Needed. 
[Hyperlink, http://www.gao.gov/cgi-bin/ getrpt?GAO-02-26] 
(Washington, D.C.: December 31, 2001). 

(450176): 

FOOTNOTES

[1] U.S. General Accounting Office, 2000 Census: Coverage Evaluation 
Interviewing Overcame Challenges, but Further Research Needed, GAO-02-
26 (Washington, D.C.: Dec. 31, 2001); and 2000 Census: Coverage 
Evaluation Matching Implemented as Planned, but Census Bureau Should 
Evaluate Lessons Learned, GAO-02-297 (Washington, D.C.: Mar. 14, 2002).

[2] GAO-04-37, pp. 43-44.

[3] Constance F. Citro, Daniel L. Cork, and Janet L. Norwood, eds., 
The 2000 Census: Counting under Adversity (Washington, D.C.: The 
National Academies Press, 2004).

[4] See GAO-02-297, pp. 9-12, for example.

[5] The Bureau uses computer-executed algorithms to estimate 
imputations.

[6] Executive Steering Committee For A.C.E. Policy II (ESCAP II) Report 
20 October 11, 2001 ESCAP II: Person Duplication in Census 2000, Thomas 
Mule, p.iv.

[7] Even so, it is not clear that all of these 1.2 million duplicates 
would have been identified had the search areas been expanded to 
completely include them, since the Bureau did not always identify 
duplicates within the search areas that it did use.

[8] Constance F. Citro, Daniel L. Cork, and Janet L. Norwood, eds., 
The 2000 Census: Interim Assessment (Washington, D.C.: The National 
Academies Press, 2001).

[9] More complete estimates of differential population undercounts are 
available on the Bureau's Internet Web site at www.census.gov. 

GAO's Mission: 

The Government Accountability Office, the investigative arm of 
Congress, exists to support Congress in meeting its constitutional 
responsibilities and to help improve the performance and accountability 
of the federal government for the American people. GAO examines the use 
of public funds; evaluates federal programs and policies; and provides 
analyses, recommendations, and other assistance to help Congress make 
informed oversight, policy, and funding decisions. GAO's commitment to 
good government is reflected in its core values of accountability, 
integrity, and reliability.

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains 
abstracts and full-text files of current reports and testimony and an 
expanding archive of older products. The Web site features a search 
engine to help you locate documents using key words and phrases. You 
can print these documents in their entirety, including charts and other 
graphics.

Each day, GAO issues a list of newly released reports, testimony, and 
correspondence. GAO posts this list, known as "Today's Reports," on its 
Web site daily. The list contains links to the full-text document 
files. To have GAO e-mail this list to you every afternoon, go to 
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order 
GAO Products" heading.

Order by Mail or Phone: 

The first copy of each printed report is free. Additional copies are $2 
each. A check or money order should be made out to the Superintendent 
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 
more copies mailed to a single address are discounted 25 percent. 
Orders should be sent to: 

U.S. Government Accountability Office

441 G Street NW, Room LM

Washington, D.C. 20548: 

To order by Phone: 

 

Voice: (202) 512-6000: 

TDD: (202) 512-2537: 

Fax: (202) 512-6061: 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470: 

Public Affairs: 

Jeff Nelligan, managing director,

NelliganJ@gao.gov

(202) 512-4800

U.S. Government Accountability Office,

441 G Street NW, Room 7149

Washington, D.C. 20548: