2010 Census:

Census Bureau Needs Procedures for Estimating the Response Rate and Selecting for Testing Methods to Increase Response Rate

GAO-08-1012: Published: Sep 30, 2008. Publicly Released: Oct 30, 2008.

Additional Materials:

Contact:

Robert N. Goldenkoff
(202) 512-6794
contact@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

The U.S. Census Bureau (Bureau) estimates that it will spend at least $2 billion to enumerate households that did not return census forms during the 2010 Census. Increasing the response rate would reduce the number of households that Bureau field staff must visit. To address concerns about reducing the cost of enumerating these households, GAO (1) analyzed how the Bureau develops, supports, and updates the response rate estimate, and the extent to which the Bureau uses the estimate to inform its 2010 planning efforts; (2) described the methods the Bureau considered for increasing response in 2010 and how it tested these methods; and (3) assessed how the Bureau identifies and selects for testing methods to increase response rate, including considering other surveys' methods. To meet these objectives, GAO analyzed the Bureau's documentation for estimating the response rate and selecting for testing methods to increase response, and interviewed experts from other survey organizations.

The 2010 Census response rate estimate is not fully supported, systematically reevaluated, or clearly incorporated into the life cycle cost estimate and planning efforts for nonresponse follow-up--where census workers visit households that do not return their census forms. Specifically, the Bureau could not demonstrate support for one component underpinning the estimate--a general decline due to decreasing public participation in surveys--because it did not document its decisions or data sources when developing the estimate. The two other estimate components that affect responses are the short-form-only census and the replacement questionnaire. In 2001, the Bureau estimated the 2010 Census response rate to be 69 percent. However, from 2001 through 2008, the Bureau did not systematically reevaluate the estimate or consider test results from this decade to determine if the estimate should be updated. Although the Bureau revised the estimate to 64 percent after a major redesign of its nonresponse follow-up operation in 2008, the Bureau still lacks procedures for establishing when and how to reevaluate and, if necessary, update the estimate. To estimate costs and plan for nonresponse follow-up, the Bureau relies on response rate estimates for local census office types because these estimates reflect geographic differences. Officials said that the local estimates reflect components of the national estimate. However, only one of the three components from the national estimate--the replacement questionnaire--was clearly reflected in the local census office type estimates. Through various national and field tests and experiments, the Bureau tested nine methods to increase 2010 Census response and currently plans to implement two of these methods--the replacement questionnaire and two-column bilingual form. The Bureau also plans to use a communications campaign to increase response and plans to test campaign messages in 2009. In July 2006, the Bureau decided not to include an Internet response option in the 2010 Census. However, the Bureau recently announced that it is again considering including the Internet option in 2010, although it has not developed further plans for testing it. For 2010, the Bureau established test objectives and research questions to identify methods to test for increasing response. However, Bureau officials did not document the methods that they considered but decided not to test or the rationale behind those decisions. Although officials said that they considered cost, complexity of the test, and compatibility of experiments in their decisions, they did not specify how they weighed these factors to select and prioritize the nine methods they chose to test. Officials said that they consider the experiences of other survey organizations to identify potential methods to increase response, but they, along with some experts, noted that such methods may only be indirectly applicable to the decennial census. Nonetheless, testing modifications to methods the Bureau has previously considered or tested, such as testing a variety of telephone reminder messages, may yield additional opportunities for increasing response.

Recommendations for Executive Action

  1. Status: Closed - Implemented

    Comments: In our 2008 report we found that the response rate for the 2010 Census was not supported by detailed documentation. For the 2020 Census the Bureau has taken steps to document the components of the response rate. For example, the initial response rate estimate for 2020 incorporated 2010 Census data, including response rates by local census office and by type of enumeration area. The response rate was then used as one of the factors the Bureau modeled for the sensitivity analysis that was used to produce the initial ROM lifecycle cost estimates for the 2020 Census.

    Recommendation: To enhance credibility of the response rate for determining cost and planning for future census activities, to inform assumptions underlying the 2020 response rate estimate, and to improve the planning and transparency of the Bureau's research and testing, the Secretary of Commerce should direct the Bureau to establish procedures for developing the 2020 response rate estimate, including documenting the data sources supporting the estimate's components and decisions that are made in establishing the components and analyzing 2010 data to assess the reasonableness of assumptions used in applying the national estimate's components to the local census offices-type (LCO) estimates.

    Agency Affected: Department of Commerce

  2. Status: Closed - Implemented

    Comments: In 2008, we reported that the Bureau did not have procedures for reevaluating and updating the 2010 response rate estimate, including identifying events or changes in related operations that should trigger a review and documenting the results of such reviews. According to the Bureau, starting in 2012 they began conducting research related to alternate response modes and to the use of administrative records to substitute for personal visit nonresponse followup; and that the results of that research and testing will inform its analysis and estimation of likely response rates and nonresponse workloads for the 2020 Census. Following that research, the Bureau will conduct specific research related to estimate 2020 response rates. In addition, response rates to the American Community Survey are being monitored throughout the decade to determine if changes in response rates are occurring. Finally, the Bureau will establish a team to monitor socio-political events that may serve as triggers for a reevaluation of the response rate estimate. According to the Bureau, if an event were to occur during intercensal testing that did warrant an examination of the response rate, then the team described above would make that determination. If the response rate were revised, the decision would be documented through a formal memo.

    Recommendation: To enhance credibility of the response rate for determining cost and planning for future census activities, to inform assumptions underlying the 2020 response rate estimate, and to improve the planning and transparency of the Bureau's research and testing, the Secretary of Commerce should direct the Bureau to establish procedures for reevaluating and updating the 2020 estimate, including identifying events or changes in related operations that should trigger a review and documenting the results of such reviews.

    Agency Affected: Department of Commerce

  3. Status: Closed - Implemented

    Comments: In our 2008 review of the Census Bureau's (Bureau) testing of methods to increase response in the 2010 Census, we found the Bureau could not specify how they selected methods to test for increasing responses; and recommended the Bureau establish procedures for selecting methods for increasing response rate that will be the subject of research and testing, including requirements for documenting how the Bureau defines and weighs factors used to select methods and documentation on methods considered but not tested. The Bureau has taken action and from 2009 to 2011 established and implemented a rigorous process for selecting research projects to test and not to test. According to Bureau documentation the Bureau cataloged over 600 ideas and has now identified 26 research projects to inform the 2020 Census design.

    Recommendation: To enhance credibility of the response rate for determining cost and planning for future census activities, to inform assumptions underlying the 2020 response rate estimate, and to improve the planning and transparency of the Bureau's research and testing, the Secretary of Commerce should direct the Bureau to establish procedures for selecting methods for increasing response rate that will be the subject of research and testing, including requirements for documenting how the Bureau defines and weighs factors used to select methods and documentation on methods considered but not tested.

    Agency Affected: Department of Commerce

 

Explore the full database of GAO's Open Recommendations »

Oct 31, 2014

Oct 30, 2014

Oct 27, 2014

Oct 24, 2014

Oct 20, 2014

Oct 9, 2014

Oct 8, 2014

Oct 2, 2014

Sep 30, 2014

Sep 26, 2014

Looking for more? Browse all our products here