Census 2000:

Design Choices Contributed to Inaccuracy of Coverage Evaluation Estimates

GAO-05-71: Published: Nov 12, 2004. Publicly Released: Dec 13, 2004.

Additional Materials:

Contact:

Robert N. Goldenkoff
(202) 512-3604
contact@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

Evaluations of past censuses show that certain groups were undercounted compared to other groups, a problem known as "coverage error." To address this, the Census Bureau included in its 2000 Census design the Accuracy and Coverage Evaluation Program (A.C.E.) to (1) measure coverage error and (2) use the results to adjust the census, if warranted. However, the Bureau found the A.C.E. results inaccurate and decided not to adjust or plan for adjustment in 2010. Congress asked GAO to determine (1) factors contributing to A.C.E.'s reported failure to accurately estimate census coverage error, and (2) the reliability of the revised coverage error estimates the Bureau subsequently produced. To do this, GAO examined three sets of Bureau research published in March 2001, October 2001, and March 2003 and interviewed Bureau officials.

According to senior Bureau officials, increasingly complicated social factors, such as extended families and population mobility, presented challenges for A.C.E., making it difficult to determine exactly where certain individuals should have been counted thus contributing to the inaccuracy of the coverage error estimates. For example, parents in custody disputes both may have an incentive to claim their child as a resident, but the Bureau used rules for determining where people should be countedresidence rules--that did not account for many of these kinds of circumstances. Other design decisions concerning both A.C.E. and the census also may have created "blind spots" that contributed to the inaccuracy of the estimates. The Bureau has not accounted for the effects of these or other key design decisions on the coverage error estimates, which could hamper the Bureau's efforts to craft a program that better measures coverage error for the next national census. Despite having twice revised A.C.E.'s original coverage error estimates, the Bureau has no reliable estimates of the extent of coverage error for the 2000 census. While both revisions suggested that the original estimates were inaccurate, in the course of thoroughly reviewing the revisions, the Bureau documented (1) extensive limitations in the revision methodology and (2) an unexpected pattern between the revised estimates and other A.C.E. data, both of which indicated that the revised coverage error estimates may be questionable themselves. Furthermore, when the Bureau published the revised estimates, it did not clearly quantify the impact of these limitations for readers, thus preventing readers from accurately judging the overall reliability of the estimates. It is therefore unclear how A.C.E. information will be useful to the public or policymakers, or how the Bureau can use it to make better decisions in the future.

Recommendations for Executive Action

  1. Status: Closed - Implemented

    Comments: According to the Department of Commerce, the developmental efforts for the Census Coverage Measurement (CCM) program have been evolving over the years leading up to the 2010 census, and won't be complete until early 2010. The documentation of these research efforts and design choices are occurring on a flow basis as various stages of work are completed. To ensure that all design changes to the Census are known to all affected teams, the Bureau has created Operational Integration Teams (OIT) around major programs, like CCM, comprising subject matter experts and representatives of each census division with responsibility for some part of the program. The OIT monitors implementation and discusses all changes needing to be made affecting its program, ensuring two-way communication to and from the OIT and the divisions with their work affected. Representatives of the CCM OIT have cited as examples of issues they have addressed: redefining and applying residence rules properly; introducing the Coverage Follow-Up operation before CCM Person Interviewing; and workarounds to obtaining map spots on military bases. The creation of these OIT teams and their prominent role in both developing and monitoring the design of census and CCM operations, combine to address this recommendation.

    Recommendation: As the Bureau plans for its coverage evaluation of the next national head count in 2010 and to ensure that coverage evaluation results the Bureau disseminates are as useful as possible to Congress and other census stakeholders, the Secretary of Commerce should direct that to avoid creating any unnecessary blind spots in the 2010 coverage evaluation, as the Bureau plans for its coverage evaluation in 2010, it should take into account how any significant future design decisions relating to census (for example, residence rules, efforts to detect and reduce duplicates, or other procedures) or A.C.E. (for example, scope of coverage, and changes in search areas, if applicable), or their interactions, could affect the accuracy of the program.

    Agency Affected: Department of Commerce

  2. Status: Closed - Implemented

    Comments: According to the Department of Commerce, specific dates on when Commerce will have a final evaluation plan for Census Coverage Measurement (CCM) have not been determined. Bureau officials have stated that they are open to conducting additional evaluations that seem warranted or appropriate as they learn more during the census as well. Nonetheless, the Bureau has committed to carrying out a series of evaluations of each of the major CCM operations and will be looking at specific changes in the design of the programs and their estimation methods. Officials mentioned the following topics as subjects of their planned evaluation reviews: changes in search areas, consistency of residence rules, changes in the CCM Person Interview operation, errors attributable to reliance on administrative records, and similarities in the wording of forms for Coverage Follow-Up and CCM Person Interviewing. Planning to evaluate such specific changes that might impact the accuracy of estimates of census coverage implements this recommendation.

    Recommendation: As the Bureau plans for its coverage evaluation of the next national head count in 2010 and to ensure that coverage evaluation results the Bureau disseminates are as useful as possible to Congress and other census stakeholders, the Secretary of Commerce should direct that, in the future, the Bureau should clearly report in its evaluation of A.C.E. how any significant changes in the design of census and/or A.C.E. might have affected the accuracy of the coverage error estimates.

    Agency Affected: Department of Commerce

  3. Status: Closed - Implemented

    Comments: Bureau executives and Census Coverage Measurement (CCM) program managers said that they were well aware of the need for being clear about the impact of limitations on evaluation results. They also noted the various challenges in itemizing all possible non-sampling errors and how expensive it can be trying to measure all of them. Within these constraints, however, the Bureau has developed an approach to improve such measurement where feasible, including a candid discussion of how the limitations could affect the utility of the evaluation results and specific information on serious flaws they uncover in their CCM methodologies. Although the CCM results are to be released after the 2010 Census, because the Bureau has a mechanism in place to improve the transparency of the limitation of the evaluation results, the Bureau's efforts address this recommendation.

    Recommendation: As the Bureau plans for its coverage evaluation of the next national head count in 2010 and to ensure that coverage evaluation results the Bureau disseminates are as useful as possible to Congress and other census stakeholders, the Secretary of Commerce should direct that, in the future, the Bureau plan to not only identify but report, where feasible, the potential range of impact of any significant methodological limitation on published census coverage error estimates. When the impact on accuracy is not readily quantifiable, the Bureau should include clear statements disclosing how it could potentially affect how people interpret the accuracy of the census or A.C.E.

    Agency Affected: Department of Commerce

 

Explore the full database of GAO's Open Recommendations »

Oct 30, 2014

Oct 27, 2014

Oct 24, 2014

Oct 20, 2014

Oct 9, 2014

Oct 8, 2014

Oct 2, 2014

Sep 30, 2014

Sep 26, 2014

Looking for more? Browse all our products here