This is the accessible text file for GAO report number GAO-11-154 
entitled '2010 Census: Follow-up Should Reduce Coverage Errors, but 
Effects on Demographic Groups Need to Be Determined' which was 
released on December 14, 2010. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as 
part of a longer term project to improve GAO products' accessibility. 
Every attempt has been made to maintain the structural and data 
integrity of the original printed product. Accessibility features, 
such as text descriptions of tables, consecutively numbered footnotes 
placed at the end of the file, and the text of agency comment letters, 
are provided but may not exactly duplicate the presentation or format 
of the printed version. The portable document format (PDF) file is an 
exact electronic replica of the printed version. We welcome your 
feedback. Please E-mail your comments regarding the contents or 
accessibility features of this document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

United States Government Accountability Office: 
GAO: 

Report to Congressional Requesters: 

December 2010: 

2010 Census: 

Follow-up Should Reduce Coverage Errors, but Effects on Demographic 
Groups Need to Be Determined: 

GAO-11-154: 

GAO Highlights: 

Highlights of GAO-11-154, a report to congressional requesters. 

Why GAO Did This Study: 

The U.S. Census Bureau (Bureau) puts forth tremendous effort to 
conduct a complete and accurate count of the nationís population and 
housing; yet some degree of error in the form of persons missed, 
duplicated, or counted in the wrong place is inevitable due to the 
complexity in counting a large and diverse population. The Bureau 
designed two operations, Coverage Follow-up (CFU) and Field 
Verification (FV), to reduce certain types of counting, or coverage, 
errors in the 2010 Census. GAO was asked to assess (1) the extent to 
which the Bureau completed CFU and FV on schedule and within estimated 
cost and (2) the implications of their key design elements for 
improving coverage. 

What GAO Found: 

The Bureau completed CFU and FV on schedule and within budget. FV cost 
$21 million (about 38 percent less than estimated) and CFU cost about 
$267 million (about 2 percent less than estimated). These operations 
followed up on potential errors on census returns or lists of 
addresses after census data had been initially collected. Their 
completion provided follow-up data used by subsequent data processing 
that removed errors from the official census tabulations. 

Three of the Bureauís key CFU design elements will likely improve 
overall census accuracy, but their effect on undercounts of different 
demographic groups is not clear. One key design element increased the 
number and types of follow-up cases. The Bureau expanded the scope of 
CFU from about 2 million households in the 2000 Census to more than 7 
million in 2010. It also added 20 different types of households for 
potential follow-up. New types included households that reported 
members temporarily residing elsewhere, such as at college, in nursing 
homes, or in jail. According to the Bureau, the 2010 CFU operation 
should remove more than 2.7 million coverage errors from the census. 

Another key design element of CFU prioritized follow-up cases based on 
their likelihood to result in a census correction, which was a 
reasonable attempt to leverage the resources for the operation. 
However, the Bureauís evaluation plans, based on considerations of 
what may best reduce cost or increase accuracy in the future, do not 
link the demographic characteristics of households to how they 
responded to the additional questions or CFU results for those 
households. Therefore, it is unclear whether the prioritized follow-up 
will help reduce differences in the accuracy of census counts across 
demographic groups. 

Finally, CFUís design relied on a telephone-only approach to complete 
follow-up rather than personal visits. This limited costs, resulting 
in more follow-up and likely more coverage errors being removed from 
the census. But the telephone-only decision excluded about 700,000 
households from CFU that could not be contacted by telephone. Prior 
Bureau experience indicates that some historically undercounted groups 
were less likely to be reachable by telephone, and more recent 
independent research suggests that trends in telecommunication usage 
may also make it harder to reach some demographic groups this way in 
the future. Yet the Bureauís evaluation plans do not include an 
assessment of either the usefulness of the telephone numbers it 
collected in reaching specific groups or the effect of these trends. 
Greater understanding of how best to reach different groups as well as 
the influence of trends on the effectiveness of CFU could help to 
control costs while working to further reduce differential undercounts. 

What GAO Recommends: 

GAO recommends that the Secretary of Commerce direct the Bureau to 
assess (1) how well questions to help identify miscounted people on 
census forms helped reduce differences in the undercounts between 
demographic groups; (2) the degree to which telephone numbers led to 
completed contacts for households of various demographic 
characteristics; and (3) how trends in telecommunication usage and new 
technology may influence the effectiveness of CFU. The Secretary of 
Commerce concurred with our recommendations. 

View [hyperlink, http://www.gao.gov/products/GAO-11-154] or key 
components. For more information, contact Robert Goldenkoff at (202) 
512-2757 or goldenkoffr@gao.gov. 

[End of section] 

Contents: 

Letter: 

Background: 

The Bureau Completed FV and CFU on Schedule and within Budget, but 
Needs to Improve the Accuracy of Its Cost Estimates: 

CFU Should Help Improve Overall Census Coverage, but Its Effect on 
Different Demographic Groups May Not Be Uniform: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments and Our Evaluation: 

Appendix I: Comments from the Department of Commerce: 

Related GAO Products: 

Table: 

Table 1: CFU Performance Exceeded Targets: 

Figure: 

Figure 1: The Bureau Used Probes on Census Questionnaires to Identify 
Households with Possible Coverage Errors: 

Abbreviations: 

Bureau: U.S. Census Bureau: 

CFU: Coverage Follow-up: 

FV: Field Verification: 

IT: information technology: 

[End of section] 

United States Government Accountability Office: 
Washington, DC 20548: 

December 14, 2010: 

The Honorable Thomas R. Carper: 
Chairman The Honorable John McCain: 
Ranking Member: 
Subcommittee on Federal Financial Management, Government Information, 
Federal Services, and International Security: 
Committee on Homeland Security and Governmental Affairs: 
United States Senate: 

The Honorable Darrell E. Issa: 
Ranking Member: 
Committee on Oversight and Government Reform: 
House of Representatives: 

The Honorable William Lacy Clay: 
Chairman: 
The Honorable Patrick T. McHenry: 
Ranking Member: 
Subcommittee on Information Policy, Census, and National Archives: 
Committee on Oversight and Government Reform: 
House of Representatives: 

The U.S. Census Bureau (Bureau) puts forth tremendous effort to 
conduct a complete and accurate count of the nation's population and 
housing; nonetheless, some degree of error in the form of persons 
missed, duplicated, or counted in the wrong place during the decennial 
census is inevitable due to the complexity of counting a large and 
diverse population. 

The Bureau estimates that the 2000 Census undercounted certain 
population groups, including minorities, renters, and children, but 
somewhat overcounted the population as a whole. An undercount occurs 
when the census misses an individual who should have been enumerated; 
an overcount occurs when an individual is counted in error. 
Differences among undercounts of ethnic, racial, and other groups are 
referred to as "differential undercounts," which may have implications 
for political representation and other uses of census data. In an 
effort to improve accuracy and reduce differential undercounts of the 
population in 2010, the Bureau conducted the Coverage Follow-up (CFU) 
and Field Verification (FV) operations, two programs intended to clean 
up possible errors identified after households provided their census 
responses. During CFU, a contractor telephoned certain households in 
an attempt to determine if someone had been miscounted, such as when 
the number of people reported living in a household did not match the 
number of people whose name and demographic information was included 
on the household's census form. During FV, the Bureau visited 
addresses that had been provided by persons that thought they had been 
missed by the census and that did not match the Bureau's master 
address list. From the 2000 Census, the Bureau expanded CFU and 
allocated more than $200 million in additional funds--including $30 
million provided by the American Recovery and Reinvestment Act of 2009 
[Footnote 1]--which allowed the Bureau to include an additional 1.1 
million households within the scope of CFU. 

After reviewing the status of CFU in 2008, we recommended that the 
Bureau submit its plans for CFU to Congress and decide how it would 
conduct the operation.[Footnote 2] The Bureau did so and completed CFU 
in August 2010 and FV in September 2010. As requested, for this review 
we examined (1) the extent to which FV and CFU were completed on 
schedule and within cost estimates, and (2) key design elements of CFU 
and FV, the implications for those design elements on improving 
coverage, and possible lessons learned to the extent similar efforts 
are used in the 2020 Census. This report is one of three we are 
releasing today.[Footnote 3] The other reports focus on the Bureau's 
efforts to reach out to and enumerate hard-to-count populations, and 
efforts to complete other key census-taking activities. Both reports 
identify preliminary lessons learned, as well as potential focus areas 
for improvement. 

To meet both objectives, we assessed Bureau planning, testing, and 
schedule documents and interviewed Bureau officials to supplement and 
verify the currency and relevance of documentation obtained. For the 
first objective we also assessed the performance of CFU and FV against 
the cost, timeliness, and other metrics the Bureau used to monitor the 
operations. Additionally, for the second objective we reviewed our 
past reports and Bureau literature on known limitations of follow-up 
methods to identify key design elements and their implications. We 
also assessed Bureau study and evaluation plans. We conducted this 
performance audit from March 2010 to December 2010 in accordance with 
generally accepted government auditing standards. Those standards 
require that we plan and perform the audit to obtain sufficient, 
appropriate evidence to provide a reasonable basis for our findings 
and conclusions based on our audit objectives. We believe that the 
evidence obtained provides a reasonable basis for our findings and 
conclusions based on our audit objectives. 

Background: 

To help ensure a complete count, the Bureau had a number of operations 
aimed at capturing census data from people and households that 
otherwise might have been missed by the census. For example, the Be 
Counted program was designed to make special census questionnaires 
available to those who may not have received one, including people who 
do not have a usual residence, such as transients, migrants, and 
seasonal farm workers. The questionnaires were placed in over 38,000 
locations across the country, including libraries, convenience stores, 
and other places people might frequent. The Bureau conducted FV to 
verify the existence of new addresses provided on these questionnaires 
and through other sources that were not already on the Bureau's master 
address list. The procedures enumerators followed to verify addresses 
in 2010 were largely similar to those used in the 2000 Census. In 
2000, the Bureau visited nearly 900,000 addresses as part of its FV 
operation, verifying the addition of over 450,000 addresses to its 
address list. 

To help ensure accuracy in the population count in 2000, the Bureau 
used telephone interviews in another operation to follow up with two 
types of household responses: households too large to include all 
their members on the form and households with apparent discrepancies 
on their questionnaires, such as when the number of people reported in 
the household population box does not match the number of people whose 
name and demographic information is included on the form. The Bureau 
placed calls to these households to determine if additional persons 
might have been missed (undercounted), if persons might have been 
counted in error (overcounted), or if persons might have been counted 
in the wrong place (possibly an overcount in one place and an 
undercount in another). The Bureau followed up on over 2.5 million 
households at a cost of approximately $67 million, resulting in over 
152,000 people being added to the official census count and 
approximately 258,000 others being removed. 

In response to Census 2000 experiences and in order to help achieve a 
Bureau goal of reducing differential undercounts, the Bureau added 
questions to the 2010 Census questionnaire to better identify 
potential coverage problems. These additional questions--called 
coverage probes--were to help identify households that may have 
omitted (undercounted) persons, due to familial relationships such as 
young children and extended family residing in the household but not 
reported on the census questionnaire due to space limitations, or 
households that may have counted persons more than once (overcounted), 
due to situations where members spent time elsewhere, such as 
relatives living in nursing homes or college dormitories. The 
resulting coverage probes used on the 2010 Census questionnaire are 
shown below in figure 1. 

Figure 1: The Bureau Used Probes on Census Questionnaires to Identify 
Households with Possible Coverage Errors: 

[Refer to PDF for image: illustration] 

Informational copy of a Census Questionnaire. 

Source: GAO Presentation of Census Bureau Information. 

[End of figure] 

The Bureau Completed FV and CFU on Schedule and within Budget, but 
Needs to Improve the Accuracy of Its Cost Estimates: 

FV Was Completed on Schedule and Well Under Budget: 

The 2010 FV operation began July 21, 15 days early, and finished on 
schedule on September 8, 2010. During that time, the Bureau visited 
nearly 456,000 addresses. The Bureau's preliminary results show that 
about 49 percent of those addresses were verified as valid housing 
units, 33 percent did not exist, and 18 percent were duplicates. The 
Bureau expects to report the final results in its formal evaluations 
of FV, planned for release in July 2011. 

Completing FV on schedule was commendable, given that the Bureau had 
to make several late changes to the design of the operation as part of 
a contingency plan. Before beginning the FV operation, the information 
technology (IT) system the Bureau had expected to use to support the 
operation experienced testing and development delays. Furthermore, the 
Bureau was experiencing difficulty with this system being used to 
support other major census field operations. Therefore the Bureau 
developed and implemented a contingency plan, which substituted a 
modified version of the IT system used to support the 2010 Census 
Coverage Measurement operations and the IT system used for the 2000 
Census, changed some related procedures for shipping workload 
materials, and significantly expanded the quality-assurance program to 
mitigate the risk of introducing additional changes to the contingency 
IT system. The Bureau attributes its ability to complete FV on time to 
its aggressive monitoring of the risk that IT systems might not be 
ready, its having identified a contingency IT system in advance, and 
the small scale of the FV operation compared to other census field 
operations, which allowed for the rapid adoption of alternative 
procedures. The Bureau's timely response to IT system delays 
demonstrates the benefits of the Bureau having developed risk 
mitigation plans for significant risks, as we have recommended before. 
[Footnote 4] 

The Bureau completed FV at a cost of $21 million, 38 percent lower 
than the $33.8 million estimated for the operation. Bureau officials 
are still researching why costs were lower than expected, but their 
preliminary analysis attributes cost savings to increased productivity 
and reduced operational inputs, such as training hours and mileage. 
The final cost of the operation was unexpected considering that the 
Bureau estimated that the late changes made by the contingency plan 
would increase the cost per case and increase total costs by nearly 
$15 million for the estimated workload. Under the Bureau's original 
calculation, adding the contingency plan raised the estimated cost-per-
case to about $74, far more than the $53 per case assumed in the 
initial budget estimate or the $46 per case reported at the end of the 
operation. The Bureau has not attempted to separately identify how 
much of the final cost per case increase was attributable to the 
contingency plan or other factors. 

Achieving cost savings is a positive development. However, our prior 
work has highlighted the need for the Bureau to develop more accurate 
and rigorous cost estimates for census operations.[Footnote 5] A high- 
quality cost estimate is the foundation of a good budget, providing an 
estimate of the funding required to efficiently execute a program. 
Additionally, according to our Cost Assessment Guide, a cost estimate 
should be a "living" document that is continually updated as actual 
costs begin to replace original estimates, so that it remains relevant 
and current.[Footnote 6] We have previously recommended[Footnote 7] 
that the Bureau document where actual costs differ from those 
estimated to help document lessons learned and the basis for changes 
made to assumptions used. The Bureau updated its cost estimate for FV 
as its estimated workload changed and when it considered adoption of 
the contingency plan. However, moving forward it will be important to 
be able to explain the variation in its cost estimates. 

CFU Was Completed on Schedule and Exceeded Performance Targets: 

The 2010 CFU operation began and finished on schedule, ensuring that 
CFU data were available for subsequent data processing. From April 11 
to August 14, Bureau contractors telephoned nearly 7.4 million 
households, of which 4.9 million (66 percent) were complete 
interviews, meeting their estimated workload target. The Bureau 
expects to be able to report an analysis of the effect of CFU on 
census coverage in the spring of 2011. 

The Bureau completed CFU at an estimated cost of $267 million, about 2 
percent less than the initial cost estimate for the operation. As 
shown in table 1 below, the Bureau completed a slightly higher 
percentage of cases it attempted and spent on average less time on the 
phone completing each interview than expected. 

Table 1: CFU Performance Exceeded Targets: 

Performance measure: Case-completion rate (percent); 
Target: 65; 
Actual: 66. 

Performance measure: Cases completed per hour per interviewer; 
Target: 2.36; 
Actual: 2.35. 

Performance measure: Average interviewer score for quality assurance 
(percent); 
Target: 97; 
Actual: 99. 

Performance measure: Average call time (seconds); 
Target: 249; 
Actual: 208. 

Source: Census Bureau. 

Note: Data are from Census Bureau management reporting. 

[End of table] 

Given mandated deadlines that the Bureau faces for delivering census 
tabulations, completing field data collection including CFU and FV on 
schedule was crucial for subsequent processing activities to proceed 
and be completed on schedule. 

CFU Should Help Improve Overall Census Coverage, but Its Effect on 
Different Demographic Groups May Not Be Uniform: 

Three design decisions for the 2010 CFU operation should improve 
overall census accuracy. These decisions include: 

* expanding the scope of the 2010 CFU operation compared to its 2000 
operation, 

* prioritizing types of follow-up cases, and: 

* contacting identified CFU households using only the telephone. 

The Bureau's Increase in the Number and Types of CFU Cases Should 
Improve Overall Census Accuracy: 

A key design decision the Bureau made for the 2010 CFU was to expand 
the scope of its coverage follow-up operation from 2000 to follow up 
on additional types of cases that it believed would help reduce the 
differential undercount. While continuing to follow up on the two 
types of cases that constituted the 2000 follow-up operation--large 
households and those with apparent count discrepancies--the Bureau 
identified 20 additional types of cases potentially to cover in 2010. 
One of these types of cases came from the use of administrative 
records from other federal sources, such as demographic information 
and addresses of families receiving tenant subsidies from an 
information system at the Department of Housing and Urban Development, 
to identify persons associated with a household's address who might 
have been omitted from the household's census form. Most of the 
additional types of cases were identified from responses to the new 
coverage probes on the census questionnaire discussed earlier. 
Additional types included households with responses indicating 
household members present who may not have been recorded on the 
initial census form, such as newborn babies, foster children, 
roommates or live-in babysitters, or household members who may have 
lived or stayed at more than one place, such as college students or 
nursing-home residents. 

Expanding the scope of the 2010 CFU increased the number of cases the 
Bureau followed up on from about 2.5 million cases in 2000 to about 8 
million cases in 2010. According to the Bureau, the 2010 CFU operation 
should result in more than 2.9 million coverage errors being removed 
from the census, including overcounts and undercounts, compared to 
more than 400,000 coverage errors being removed in 2000. The more 
coverage errors are removed from the official census count, the more 
the overall accuracy of the census is improved. 

The Effect of the Bureau's Prioritization of Coverage Follow-up on 
Differential Undercounts Is Not Clear: 

A second key design decision the Bureau made for CFU was to prioritize 
the types of cases it would follow up on, likely increasing the 
overall number of coverage errors corrected but possibly affecting 
demographic groups differently. The Bureau had to prioritize follow-up 
cases, since the expansion of the CFU scope resulted in an estimated 
22.5 million potential CFU cases, far exceeding what its time and 
budget for the 2010 CFU operation would permit. Bureau planning 
documents indicate that along with the extra time needed to pursue 
22.5 million cases, it would have cost approximately $800 million more 
to complete all of the possible follow-up cases. 

In April 2010, the Bureau formally documented its decision to 
prioritize follow-up based on cost and estimates of the number of 
corrections that would result from each type of case, and documented 
which cases it planned to include in follow-up. The Bureau estimated 
that in addition to households with apparent discrepancies on their 
questionnaires and households too large to include all their members 
on the form--the two types of cases it focused on in 2000--it would 
complete follow-up on 8 of 20 additional types of cases. However, due 
to a combination of higher call efficiency and lower-than-expected 
workloads for some of the selected types of cases, in the end the 
Bureau was able to follow up on an additional three types of CFU cases. 

The Bureau's decisions to expand the scope of CFU and prioritize the 
CFU cases will likely result in a greater number of coverage errors 
being removed from official census tabulations than were removed in 
2000, increasing CFU's effectiveness in improving overall census 
accuracy in 2010. Prioritization of the CFU cases to contact was a 
reasonable attempt to leverage the resources and time available for 
the operation. However, because the specific coverage probes the 
Bureau used on census questionnaires do not clearly map to specific 
demographic groups, the extent to which the prioritized follow-up will 
help achieve the Bureau's goal of reducing the differential undercount 
is unclear. For example, one of the Bureau's priorities for follow-up 
was households indicating that persons may have been included who 
should have been counted elsewhere, such as persons in jail, in 
college, or in the military. Among the types that the Bureau did not 
follow up on were households that reported persons who were sometimes 
living elsewhere seasonally or as part of custody-sharing 
arrangements. The demographic characteristics of the people covered by 
each of these probes are not likely to be the same as the general 
population. Thus, following up on one set of cases but not others will 
likely have a different, though unknown, effect on particular 
demographic groups. 

The Bureau acknowledges that, in so far as households in certain 
groups are not contacted by telephone but have coverage errors, there 
would likely be a smaller improvement in coverage for those 
demographic groups. Yet Bureau officials maintain that since it would 
be difficult to design follow-up based directly on household 
characteristics such as race and Hispanic origin, relying on situation-
and relationship-based probes on census questionnaires--such as the 
Bureau did in 2010--may be the most effective way to identify coverage 
follow-up cases. To that end, the Bureau has 2010 data on how each 
household responded to each coverage probe and the demographics of 
each household, and will ultimately know which probes led to 
corrections of census coverage errors. The Bureau plans to report the 
demographic groups for which CFU identified corrections as part of its 
formal assessment of CFU, but that assessment does not include a link 
between the results for each probe and historically undercounted 
groups. Bureau officials have explained that its evaluation choices 
are driven by consideration of the best value to the decennial census 
in terms of informing possible cost benefit analyses and improvements 
to accuracy, and can change in response to new information the Bureau 
may obtain. Linking these data from 2010 CFU could help the Bureau 
decide which of the probes and priorities best helped the Bureau both 
improve overall accuracy and reduce the differential undercount, and 
better inform the Bureau decision making on the use of coverage probes 
for 2020. 

The Effect of the Bureau's Telephone-Only Approach on Differential 
Undercounts Is Not Clear: 

A third key design decision the Bureau made for CFU was to rely solely 
on telephone interviews rather than use personal visits to contact 
households during the 2010 CFU operation. For those households the 
Bureau can reach by telephone, this decision should have enabled the 
Bureau to reduce many more census errors, because it allowed the 
Bureau to follow up with more households than it could have with 
personal visits using the same level of funding. A Bureau evaluation 
of its 2000 coverage follow-up activities suggested that personal 
visits be used for households for which the Bureau has no valid 
telephone number, noting that traditionally undercounted groups, such 
as renters and certain minority groups, were less likely to have valid 
phone numbers. Yet personal visits are significantly more expensive 
than telephone calls, costing about $71 per case compared to about $19 
per case for telephone calls, according to Bureau results from a 2006 
census test of CFU. 

To increase the effectiveness of its telephone-only approach, the 
Bureau implemented several specific recommendations from its 
evaluation of the 2000 follow-up operation, including the use of a 
commercial database to assist with identifying the telephone numbers 
that were invalid or missing for CFU cases. Bureau officials believe 
that adopting these recommendations led to significant increases in 
the effectiveness of telephone follow-up, with reported response rates 
to telephone calls conducted as part of 2010 Census tests of 63 
percent in 2004, 80 percent in 2005, and 78 percent in 2006. For 2010 
CFU, Bureau management reporting indicates that the response rate 
exceeded its target goal of 65 percent. 

The Bureau's decision to rely solely on telephone calls and related 
measures to complete CFU resulted in lower cost and more effective 
follow-up, and should contribute to a greater number of coverage 
errors being removed from official census tabulations compared to 
2000. While these design decisions helped increase overall census 
accuracy, their effect on the Bureau's goal to reduce the differential 
undercount is less clear. This is because the Bureau excluded from CFU 
any household for which it did not obtain a useable telephone number. 
A useable phone number was obtained either from the respondent's 
return and could be a wireless or landline telephone number or through 
a telephone number look-up that resulted in the retrieval of landline 
numbers only. This is potentially problematic for two reasons. 

First, current research indicates that there are significant 
differences between households that use only wireless telephones and 
those that have landline telephones, with groups of households with 
high wireless-only usage being part of historically undercounted 
populations.[Footnote 8] According to our analysis of Bureau 
management reporting, the Bureau excluded about 700,000 households 
from follow-up because it lacked a usable contact telephone number. 
The Bureau relied only on landline telephone numbers from its 
commercial database, due to concerns about not knowing the geographic 
location of wireless phones it might dial and the possible financial 
burden on wireless customers from unsolicited calls. The Bureau did 
not immediately know which household-provided numbers were wireless, 
so it had rules concerning when calls could be made, to restrict calls 
to local times appropriate for the location of addresses provided. The 
Bureau has not attempted to track how many of the telephone numbers it 
called or excluded from follow-up were wireless numbers, what types of 
CFU cases they were for, or what the demographic characteristics of 
these households might be. According to the Bureau, it has asked the 
contractor that carried out the telephone calls for the Bureau to 
report the disposition of cases where a number was found during look-
up as part of the contractor's forthcoming assessment. Future Bureau 
decisions about how to contact households for follow-up can be better 
informed if the Bureau strengthens its understanding of how different 
sources of contact information can affect its goal to reduce the 
differential undercount. 

Furthermore, trends regarding the use of wireless telephones indicate 
that some households within hard-to-count populations may be harder to 
contact in the future using landline telephone operations. For 
example, a recent Centers for Disease Control and Prevention study 
shows that wireless-only households has more than doubled between 
January 2006 and December 2009 from about 11 percent to more than 24 
percent of the nation's households. And in May 2010, the Pew Research 
Center found that wireless-only adults pose a significant challenge to 
data collection by telephone, because such adults are substantially 
different demographically from those reached on a landline phone. 
According to the Pew Research Center, wireless-only adults tend to be 
young, single, renters, and with lower income. This research also 
found that minorities made up a larger share of the wireless-only 
group with far more Hispanics, African-Americans, and people of other 
or mixed racial backgrounds than those with landline telephones. Such 
trends could pose a challenge to possible future Bureau reliance on 
telephone-based contacts intended to help improve census coverage of 
such demographic groups to the extent they do not provide their 
telephone numbers on census questionnaires. Broader ongoing shifts in 
the use of other telecommunications, including a variety of social 
media, may also influence the relative effectiveness of strategies 
relying on telephone communication. 

Conclusions: 

Overall, the Bureau generally implemented FV and CFU on schedule and 
under budget, which is a noteworthy accomplishment given the inherent 
challenges of conducting a cost-effective census. The Bureau also 
updated its cost estimates for FV periodically as its estimated 
workload for FV evolved, and adopted design changes for CFU that 
likely contributed significantly to improving the overall accuracy of 
the census. At the same time, the Bureau's experience in implementing 
these two operations highlights additional actions that may improve 
the Bureau's understanding of the effectiveness of CFU in reducing 
differential undercounts and help improve planning for 2020 to the 
extent that the Bureau conducts such operations as part of the next 
decennial. 

First, our previous work has highlighted the importance of accurate 
and rigorous cost estimates; thus, it will be important for the Bureau 
to assess the factors that led to significant variance in cost 
estimates for FV. Knowing this will allow the Bureau to develop more 
accurate cost estimates in the future, and will help the Bureau focus 
on cost containment as it prepares for the 2020 Census. We are not 
making recommendations at this time regarding the Bureau's cost 
estimation methods, as we have already done so in a previous report. 
[Footnote 9] The Bureau agreed with those recommendations at that time 
and has begun implementing them. 

Second, given the research and testing of coverage probes leading up 
to their use in 2010, it will be important for the Bureau to assess 
the degree to which the coverage probes helped address the 
differential undercount. This will help the Bureau understand whether 
the probes the Bureau prioritized for follow-up worked as intended and 
could help the Bureau determine which probes or other follow-up 
procedures to use in the future to improve census accuracy. 

Third, it will be important for the Bureau to determine the 
effectiveness of the phone numbers it obtained from census returns or 
its commercial database in making contact with households of different 
follow-up and demographic groups. The decision to rely solely on 
telephone calls and no personal visits involved an implicit trade-off 
between the opportunity for cost savings with improved overall census 
accuracy and an opportunity possibly to help reduce historic 
differential undercounts. If the Bureau better understood the 
demographic composition of those for whom it did and did not obtain 
telephone numbers, and for whom it was and was not successful in 
reaching by phone, it could better determine the effect of this design 
decision on differential undercounts. Also, better knowledge of how 
best to reach different groups could help identify effective sources 
of contact information or strategies for using them for future 
censuses, further helping to control costs while still working to 
address differential undercounts. 

Finally, whether it is a trend of households moving away from reliance 
on landlines, or other possible emerging trends related to growth in 
other modes of communication and new technology, the Bureau's future 
design decisions will benefit from tracking and assessing the 
implications of such trends and leveraging or mitigating their effect 
where possible. For example, under a scenario where the Bureau 
continues to rely on telephones for contacting households, the Bureau 
might need to adopt strategies for increasing the number of usable 
telephone numbers provided by census respondents or revisit its 
specific rules concerning when to dial numbers. Moving forward, it 
will be important for the Bureau to identify how rapid changes in 
technology and the public's use of them may affect the effectiveness 
of its efforts to improve census accuracy, both overall and in terms 
of reducing differential undercounts. 

Recommendations for Executive Action: 

We recommend that the Secretary of Commerce require the Director of 
the U.S. Census Bureau to take the following three actions to improve 
the Bureau's planning for the 2020 Census: 

To help the Bureau decide which coverage probes, if any, to use and 
prioritize for future follow-up efforts, assess the extent to which 
historically overcounted and undercounted demographic groups responded 
to the probes the Bureau followed up on and determine the 
effectiveness of specific probes in reducing differential undercounts. 

* To support the Bureau's efforts to control costs while improving 
census accuracy, determine the demographic characteristics of the 
households for which it did and did not obtain telephone numbers and, 
to the extent feasible, assess the degree to which the telephone 
numbers were usable and led to completed contacts for households of 
various follow-up groups and demographic characteristics. 

* To ensure that the design of future follow-up efforts is effective 
in improving census coverage, assess the implications that trends in 
landline and wireless usage and other modes of communication and new 
technology may have both on the design decisions for future CFU-like 
operations and on their effectiveness in improving census coverage in 
terms of both overall census accuracy and differential undercounts. 

Agency Comments and Our Evaluation: 

The Secretary of Commerce provided written comments on a draft of this 
report on December 1, 2010. The comments are reprinted in appendix I. 
The Department of Commerce agreed with the overall findings and 
recommendations and appreciated our efforts in helping the Bureau 
develop a successful evaluation plan for the 2020 Census. The 
department also included comments from the Bureau on certain 
statements in the report. 

The Bureau commented on our discussion of its initial estimate that 
the contingency plan it adopted would increase the cost of FV. The 
Bureau commented that its initial estimate that the contingency plan 
would increase the cost of FV by $15 million was based on estimated 
workloads, that the final FV workload was much smaller, and that it 
had not attempted to reestimate the cost effect of the contingency 
plan separately. We revised the text to more fully reflect that the 
estimated increase was based on estimated workload, and emphasized the 
changes in "cost per case," which better reflects the effect on cost 
of changes in workload. 

The Bureau also commented on our discussion of telecommunication 
trends and the Bureau's need to understand how different sources of 
contact information can affect its goal to reduce the differential 
undercount. The Bureau agreed that the trend toward wireless 
communication needs more attention in the future and described how it 
had not yet collected certain data that might be needed to carry out 
evaluation of the type we recommended. According to our analyses, the 
additional data are easily obtained. We revised the text to point more 
specifically to the type of data that could help the Bureau with 
future decisions about how to reach historically undercounted groups. 

We are sending copies of this report to the Secretary of Commerce, the 
Director of the U.S. Census Bureau, and interested congressional 
committees. The report also is available at no charge on GAO's Web 
site at [hyperlink, http://www.gao.gov]. 

If you have any questions about this report please contact me at (202) 
512-2757 or goldenkoffr@gao.gov. Contact points for our Offices of 
Congressional Relations and Public Affairs may be found on the last 
page of this report. Key contributors to this report were Ty Mitchell, 
Assistant Director; Mark Abraham; Sara Daleksi; Ron Fecso; Andrea 
Levine; Donna Miller; Jessica Thomsen; Jonathon Ticehurst; Holly 
Williams; and Katherine Wulff. 

Signed by: 

Robert Goldenkoff: 
Director: Strategic Issues: 

[End of section] 

Appendix I: Comments from the Department of Commerce: 

Note: Page numbers in the draft report may differ from those in this 
report. 

The Secretary Of Commerce: 
Washington, D.C. 20230: 

December 1, 2010: 

Mr. Robert Goldenkoff: 
Director: 
Strategic Issues: 
United States Government Accountability Office: 
Washington, DC 20548: 

Dear Mr. Goldenkoff: 

The U.S. Department of Commerce appreciates the opportunity to comment 
on the draft report by the United States Government Accountability 
Office (GAO), entitled "2010 Census: Follow-up Should Reduce Coverage 
Errors, but Effects on Demographic Groups Need to be Determined" (GA0-
11-154). The Department's comments on this report are enclosed.
Enclosure 

Sincerely, 

Signed by: 

Gary Locke: 

Enclosure: 

[End of letter] 

U.S. Department of Commerce Comments on the United States Government 
Accountability Office Draft Report, Entitled "2010 Census: Follow-up 
Should Reduce Coverage Errors, but Effects on Demographic Groups Need 
to be Determined" (GA0-11-154); December 2010. 

The U.S. Department of Commerce thanks the U.S. Government 
Accountability Office (GAO) for its efforts in examining the 2010 
Census Coverage Follow-up (CFU) and Field Verification (FV) 
operations. The GAO assessed the Census Bureau's performance in 
completing these operations on schedule and within budget and 
evaluated their efficacy in accurately reducing coverage errors. 

We have no fundamental disagreements with the overall findings or with 
the recommendations regarding items to be studied for the 2020 Census. 
However, the Census Bureau provides the following comments about 
statements and conclusions in this report. 

* Page 7, first full paragraph: "The final cost of the operation was 
unexpected considering that the Bureau estimated that the late changes 
made by the contingency plan would increase costs by nearly $15 
million." 

Census Bureau response: Prior to the operation, the Census Bureau used 
a preliminary workload estimate to roughly calculate operational costs 
under the contingency plan. Using the preliminary workload figures, 
the estimated impact was a $15 million increase in the budget. The 
actual workload was much lower than originally estimated. The total 
budget for the entire operation was only $33.8 million, including any 
impact of the contingency approach. The Census Bureau did not separate 
out the costs associated with the contingency plan, though clearly it 
would be less than the earlier $15 million estimate because the actual 
workload was much smaller than what was assumed when constructing that 
earlier impact estimate. 

* Page 16, paragraph continued from page 15: "The Bureau has not 
attempted to track how many of the households it called or excluded 
from follow-up had wireless numbers, what types of CFU cases they 
were, or what the demographic characteristics of these households 
might be." 

Census Bureau response: The Census Bureau agrees the trend towards 
wireless numbers (and away from landlines) needs more attention in the 
future for operations like CFU that depend entirely on telephone 
interviews. Unfortunately, the Census Bureau did not collect the data 
during the 2010 Census CFU operation that would be needed to conduct 
the analysis GAO recommends. The numbers that we called during CFU 
came primarily from the telephone numbers that people provided to us 
on their 2010 Census questionnaire. Some of the numbers would have 
been for landline telephones, and some numbers would have been for 
wireless telephones. However, we didn't collect the information to 
know which numbers corresponded to which devices, or whether people we 
called on a landline telephone also had a wireless telephone number. 
For the 2020 Census research and testing program, the Census Bureau is 
committed to closely following and studying these trends towards 
wireless communications. 

In conclusion, we acknowledge the GAO's extensive work in reviewing 
these activities, and we appreciate its ongoing efforts to help us 
develop a successful evaluation plan for the 2020 Census. 

[End of section] 

Related GAO Products: 

2010 Census: Key Efforts to Include Hard-to-Count Populations Went 
Generally as Planned; Improvements Could Make the Efforts More 
Effective for Next Census. [hyperlink, 
http://www.gao.gov/products/GAO-11-45]. Washington, D.C.: December 14, 
2010. 

2010 Census: Data Collection Operations Were Generally Completed as 
Planned, but Long-standing Challenges Suggest Need for Fundamental 
Reforms. [hyperlink, http://www.gao.gov/products/GAO-11-193]. 
Washington, D.C.: December 14, 2010. 

2010 Census: Census Bureau Continues to Make Progress in Mitigating 
Risks to a Successful Enumeration, but Still Faces Various Challenges. 
[hyperlink, http://www.gao.gov/products/GAO-10-132T]. Washington, 
D.C.: October 7, 2009. 

GAO Cost Estimating and Assessment Guide: Best Practices for 
Developing and Managing Capital Program Costs. [hyperlink, 
http://www.gao.gov/products/GAO-09-3SP]. Washington, D.C.: March 2009. 

2010 Census: The Bureau's Plans for Reducing the Undercount Show 
Promise, but Key Uncertainties Remain. [hyperlink, 
http://www.gao.gov/products/GAO-08-1167T]. Washington, D.C.: September 
23, 2008. 

2010 Census: Census Bureau Should Take Action to Improve the 
Credibility and Accuracy of Its Cost Estimate for the Decennial 
Census. [hyperlink, http://www.gao.gov/products/GAO-08-554]. 
Washington, D.C.: June 16, 2008. 

2010 Census: Bureau Needs to Specify How It Will Assess Coverage 
Follow-up Techniques and When It Will Produce Coverage Measurement 
Results. [hyperlink, http://www.gao.gov/products/GAO-08-414]. 
Washington, D.C.: April 15, 2008. 

Information Technology: Census Bureau Needs to Improve Its Risk 
Management of Decennial Systems. [hyperlink, 
http://www.gao.gov/products/GAO-08-79]. Washington, D.C.: October 5, 
2007. 

[End of section] 

Footnotes: 

[1] Pub. L. No. 111-5, div. A, tit. II, 123 Stat. 115, 127. 

[2] GAO, 2010 Census: Bureau Needs to Specify How It Will Assess 
Coverage Follow-up Techniques and When It Will Produce Coverage 
Measurement Results, [hyperlink, 
http://www.gao.gov/products/GAO-08-414] (Washington, D.C.: Apr. 15, 
2008). 

[3] GAO, 2010 Census: Key Efforts to Include Hard-to-Count Populations 
Went Generally as Planned; Improvements Could Make the Efforts More 
Effective for Next Census, [hyperlink, 
http://www.gao.gov/products/GAO-11-45] (Washington, D.C.: Dec. 14, 
2010) and GAO, 2010 Census: Data Collection Operations Were Generally 
Completed as Planned, but Long-standing Challenges Suggest Need for 
Fundamental Reforms, [hyperlink, 
http://www.gao.gov/products/GAO-11-193] (Washington, D.C.: Dec. 14, 
2010). 

[4] GAO, Information Technology: Census Bureau Needs to Improve Its 
Risk Management of Decennial Systems, [hyperlink, 
http://www.gao.gov/products/GAO-08-79] (Washington, D.C.: Oct. 5, 
2007). 

[5] GAO, 2010 Census: Census Bureau Should Take Acton to Improve the 
Credibility and Accuracy of Its Cost Estimate for the Decennial 
Census, [hyperlink, http://www.gao.gov/products/GAO-08-554] 
(Washington, D.C.: June 16, 2008). 

[6] GAO, GAO Cost Estimating and Assessment Guide: Best Practices for 
Developing and Managing Capital Program Costs, [hyperlink, 
http://www.gao.gov/products/GAO-09-3SP] (Washington, D.C.: March 2009). 

[7] [hyperlink, http://www.gao.gov/products/GAO-08-554]. 

[8] National Center for Health Statistics, Wireless Substitution: 
Early Release of Estimates from the National Health Interview Survey, 
July-December 2009 (May 2010). Pew Research Center, Assessing the Cell 
Phone Challenge to Survey Research in 2010 (May 2010). 

[9] [hyperlink, http://www.gao.gov/products/GAO-08-554]. 

[End of section] 

GAO's Mission: 

The Government Accountability Office, the audit, evaluation and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the performance 
and accountability of the federal government for the American people. 
GAO examines the use of public funds; evaluates federal programs and 
policies; and provides analyses, recommendations, and other assistance 
to help Congress make informed oversight, policy, and funding 
decisions. GAO's commitment to good government is reflected in its core 
values of accountability, integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each 
weekday, GAO posts newly released reports, testimony, and 
correspondence on its Web site. To have GAO e-mail you a list of newly 
posted products every afternoon, go to [hyperlink, http://www.gao.gov] 
and select "E-mail Updates." 

Order by Phone: 

The price of each GAO publication reflects GAOís actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAOís Web site, 
[hyperlink, http://www.gao.gov/ordering.htm]. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional 
information. 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: 
E-mail: fraudnet@gao.gov: 
Automated answering system: (800) 424-5454 or (202) 512-7470: 

Congressional Relations: 

Ralph Dawn, Managing Director, dawnr@gao.gov: 
(202) 512-4400: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7125: 
Washington, D.C. 20548: 

Public Affairs: 

Chuck Young, Managing Director, youngc1@gao.gov: 
(202) 512-4800: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7149: 
Washington, D.C. 20548: