2010 Census:

Basic Design Has Potential, but Remaining Challenges Need Prompt Resolution

GAO-05-9: Published: Jan 12, 2005. Publicly Released: Feb 11, 2005.

Additional Materials:

Contact:

Robert N. Goldenkoff
(202) 512-3604
contact@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

A rigorous testing and evaluation program is a critical component of the census planning process because it helps the U.S. Census Bureau (Bureau) assess activities that show promise for a more cost-effective head count. The Bureau conducted a field test in 2004, and we were asked to (1) assess the soundness of the test design and the extent to which the Bureau implemented it consistent with its plans, (2) review the quality of the Bureau's information technology (IT) security practices, and (3) identify initial lessons learned from conducting the test and their implications for future tests and the 2010 Census.

The Bureau's design for the 2004 census test addressed important components of a sound study, and the Bureau generally implemented the test as planned. For example, the Bureau clearly identified its research objectives, developed research questions that supported those objectives, and developed evaluation plans for each of the test's 11 research questions. The initial results of the test suggest that while certain new procedures show promise for improving the cost-effectiveness of the census, the Bureau will have to first address a number of problems that could jeopardize a successful head count. For example, enumerators had little trouble using hand held computers (HHC) to collect household data and remove late mail returns. The computers could reduce the Bureau's reliance on paper questionnaires and maps and thus save money. The test results also suggest that certain refinements the Bureau made to its procedures for counting dormitories, nursing homes, and other "group quarters" could help prevent the miscounting of this population group. Other aspects of the test did not go as smoothly. For example, security practices for the Bureau's IT systems had weaknesses; the HHCs had problems transmitting data; questionnaire items designed to improve coverage and better capture race/ethnicity confused respondents; enumerators sometimes deviated from prescribed enumeration procedures; and certain features of the test were not fully operational at the time of the test, which hampered the Bureau from fully gauging their performance. With few testing opportunities remaining, it will be important for (1) the Bureau to find the source of these problems, devise cost-effective solutions, and integrate refinements before the next field test scheduled for 2006, and (2) Congress to monitor the Bureau's progress in resolving these issues.

Recommendations for Executive Action

  1. Status: Closed - Implemented

    Comments: In the 2004 census test, the Bureau conducted a targeted second mailing of the census questionnaire to those households that did not return the original questionnaire sent to them. However, the targeted second mailing was not one of the test objectives, and its feasibility and impact were not formally assessed. Throughout the engagement we discussed the Bureau's plans for the targeted second mailing and recommended that the Bureau analyze the targeted second mailing to determine its operational feasibility and whether cost savings could be achieved. The Bureau implemented our recommendation in February 2005 when a contractor provided the Bureau with an assessment of targeted second mailing to determine whether such an endeavor was operationally feasible for 2010. The contractor found that printing and mailing technology was available, thus making it feasible to implement a successful replacement mailing program for the 2010 Census. Based on the contractor study, upwards of a billion dollars can be saved as a result of not having to conduct non-response follow-up activities for those households that return the replacement form. Finally, the Bureau has also made targeted replacement mailing one of its test objectives for the 2006 Census Test. The Bureau also implemented our recommendation to analyze the impact the HHC had on cost. On September 29, 2005, the Bureau issued evaluations on the use of the HHC in the 2004 test. The evaluations from 2004 show the impact that HHC has on cost. Specifically, the evaluations found that enumerator productivity for collecting non-response follow-up data was reduced, that the HHC did not save office space or reduce staffing.

    Recommendation: To facilitate effective census planning and development, and to help the Bureau achieve its key goals for the census--reduce risks, improve accuracy, and contain costs, the Secretary of Commerce should direct the Bureau to analyze the impact that handheld computers (HHCs) and the targeted second mailing had on cost savings and other Bureau objectives.

    Agency Affected: Department of Commerce

  2. Status: Closed - Implemented

    Comments: To help ensure that the Bureau's IT systems were properly protected against intrusion or unauthorized disclosure of sensitive information, we recommended the Bureau ensure that future IT systems be in full compliance with FISMA and the Bureau's own internal IT policies. In response to our recommendation, the Bureau has taken steps to strengthen its IT security risk management practices. Specifically, the Bureau's Assistant Director for Decennial IT and Geographic Systems, in a January 26, 2005, memorandum to all of the decennial census division chiefs, outlined a set of enhanced IT security practices and policies that the divisions are to follow to ensure that systems developed for future tests addressed the shortcomings GAO identified, and complied with applicable laws and regulations. The practices and policies included requirements to perform a complete and thorough risk assessment, develop an inventory of all applications and general support IT systems associated with the system, and ensure that all IT security practices are in full compliance with FISMA as well as federal and Bureau IT policies.

    Recommendation: To facilitate effective census planning and development, and to help the Bureau achieve its key goals for the census--reduce risks, improve accuracy, and contain costs, the Secretary of Commerce should direct the Bureau to ensure the Bureau's IT security practices are in full compliance with applicable requirements, such as the Federal Information Security Management Act of 2002, as well as its own internal policies.

    Agency Affected: Department of Commerce

  3. Status: Closed - Implemented

    Comments: In 2004, the U.S. Census Bureau (Bureau) tested the feasibility of providing enumerators with handheld computers (HHC) to collect non-response follow-up data. In January 2005, we reported that HHCs experienced both functionality and reliability problems and we recommended the Bureau eliminate memory overloads, improve the dependability of transmissions, speed up the mapping software and make editing the questionnaire easier. The Bureau has addressed these technical problems. In April 2005, the Bureau provided us with an action plan that addressed our recommendations. For example, to eliminate overloads the Bureau upgraded the operating system that was used in 2004, and according to the Bureau, the new system is more robust and addresses system instabilities experienced by the old system. The Bureau also streamlined its in-house software design and removed a third-party product, thus allowing greater programming flexibility. The Bureau also has simplified the transmission process. According to the Bureau, during the 2004 census test, transmissions were part of the Assignment Management System (AMS), which required the enumerator to go into AMS and transmit the data. The 2006 census test software has been arranged so that transmissions occur outside of AMS. The speed of the mapping feature has also been addressed. Specifically, for the 2006 census test, the Bureau obtained compression software that provides better performance for map displays on the HHC, and it procured a faster processor and a new generation of memory cards to improve applications.

    Recommendation: To facilitate effective census planning and development, and to help the Bureau achieve its key goals for the census--reduce risks, improve accuracy, and contain costs, the Secretary of Commerce should direct the Bureau to enhance the reliability and functionality of HHCs by, among other actions, (1) improving the dependability of transmissions, (2) exploring the ability to speed up the mapping feature, (3) eliminating the causes of crashes, and (4) making it easier for enumerators to edit questionnaires.

    Agency Affected: Department of Commerce

  4. Status: Closed - Implemented

    Comments: During the 2004 census test, the U.S. Census Bureau (Bureau) tested the handheld computers (HHC) ability to (1) update the Bureau's master address list, and (2) collect data from non-responding households. However, our observations and the Bureau's test results revealed problems with the ability of the HHCs to transmit data as well as other performance issues. These shortcomings hampered the Bureau's ability to fully evaluate the performance of the HHCs. To help the Bureau achieve its key goals for the Census, we recommended the Bureau define specific, measurable performance requirements for the HHCs that address such important measures as productivity, cost savings, reliability, durability, and test their ability to meet those requirements. The Bureau has taken a number of steps to address our recommendation. In 2006, the Bureau awarded a contract for the development of a new HHC device. During the 2008 dress rehearsal, the Bureau reported on HHC performance using measures such as the number of defective and broken HHCs and in October 2008, the Bureau approved a contract modification that contained measurable performance reporting requirements for the HHC during address canvassing and tested those requirements in December 2008.

    Recommendation: To facilitate effective census planning and development, and to help the Bureau achieve its key goals for the census--reduce risks, improve accuracy, and contain costs, the Secretary of Commerce should direct the Bureau to define specific, measurable performance requirements for the HHCs and other census-taking activities that address such important measures as productivity, cost savings, reliability, durability, and test their ability to meet those requirements in 2006.

    Agency Affected: Department of Commerce

  5. Status: Closed - Implemented

    Comments: In response to our recommendation to test the wording and formatting of the coverage and race and ethnicity questions, the Bureau indicated that it had conducted cognitive tests for both questions and that it would use those results to inform further testing. In September 2005, as part of the 2005 National Census Test, the Bureau tested 5 ways of presenting the Hispanic origin, race, and ancestry question. The Bureau also continues to test how to improve coverage. One of the objectives of the 2006 test is to test coverage improvement including continued testing of residence rules and the coverage questions. This additional testing being conducted by the Bureau implements our recommendation, thus we consider this recommendation to be closed.

    Recommendation: To facilitate effective census planning and development, and to help the Bureau achieve its key goals for the census--reduce risks, improve accuracy, and contain costs, the Secretary of Commerce should direct the Bureau to review and test the wording and formatting of the coverage and race/ethnicity questions to make them less confusing to respondents and thus help ensure the collection of better quality data, and ensure they are formatted the same way on both the HHC and paper versions of the census form.

    Agency Affected: Department of Commerce

  6. Status: Closed - Implemented

    Comments: In keeping with GAO's recommendation, the Census Bureau (Bureau) has taken steps to develop a more strategic approach to training. For example, in 2007, the Bureau issued a report on its evaluation of the training program for the Nonresponse Followup (NRFU) operation. The evaluation focused on improving training delivery and content based on operational needs. Also, in 2009, the Bureau completed its 2010 enumerator training manual and instructor guide for NRFU, which updated potential concerns of reluctant respondents to include identity theft and provided more examples of strategies to deal with reluctant respondents.

    Recommendation: To facilitate effective census planning and development, and to help the Bureau achieve its key goals for the census--reduce risks, improve accuracy, and contain costs, the Secretary of Commerce should direct the Bureau to develop a more strategic approach to training by ensuring the curriculum and instructional techniques (1) are aligned with the skills and competencies needed to meet the Bureau's data collection requirements and methodology and (2) address challenges identified in the 2004 test and previous censuses.

    Agency Affected: Department of Commerce

  7. Status: Closed - Implemented

    Comments: Based on the 2004 census test, we recommended that the Bureau revisit procedures to properly locate and count persons living in group quarters. The Bureau agreed and has implemented our recommendation. For example, Bureau revised its group quarters definitions and tested them during the 2006 Census Test. Specifically, the Bureau refined group quarters procedures to clearly instruct listers regarding correcting and deleting addresses, revised definitions to ensure that group quarters are correctly identified, and that group homes are classified as group quarters and not housing units.

    Recommendation: To facilitate effective census planning and development, and to help the Bureau achieve its key goals for the census--reduce risks, improve accuracy, and contain costs, the Secretary of Commerce should direct the Bureau to revisit group quarter procedures to ensure they allow the Bureau to best locate and count this population group.

    Agency Affected: Department of Commerce

  8. Status: Closed - Implemented

    Comments: We recommended the Bureau conduct small scale testing of hand held computers (HHC) prior to the 2006 Census Test. The Bureau took a number of actions during 2005 to implement our recommendation. For example, prior to addressing canvassing for the 2006 test, the Bureau did its own three-tier testing of the software, testing: 1) the systems, 2) the integration, and 3) performance in the field through a simulation in Prince George's County, Maryland.

    Recommendation: To facilitate effective census planning and development, and to help the Bureau achieve its key goals for the census--reduce risks, improve accuracy, and contain costs, the Secretary of Commerce should direct the Bureau to ensure that all systems and other census-taking functions are as mature as possible and test ready prior to their deployment for the 2006 test, in part by conducting small-scale, interim tests under the various conditions and environments the Bureau is likely to encounter during the test and actual enumeration.

    Agency Affected: Department of Commerce

  9. Status: Closed - Implemented

    Comments: Since we issued this report, the Bureau has been updating Congress on its progress. For example, the Bureau testified on April 19, 2005, on the status of its re-engineered approach, which includes address file and map enhancements, a short form only census, and early planning and testing for 2010. In September 2005, the Bureau provided Congress with an updated 2010 life cycle cost document that discusses the components of its cost estimate for conducting the 2010 census, a document the Bureau intends to update annually. Finally, the Bureau testified on March 1, 2006, on the progress of the 2010 Census.

    Recommendation: Further, to ensure the transparency of the census-planning process and facilitate Congressional monitoring, the Secretary of Commerce should direct the Bureau to regularly update Congress on the progress it is making in addressing these and any other challenges, as well as the extent to which the Bureau is on track for meeting the overall goals of the 2010 Census.

    Agency Affected: Department of Commerce

 

Explore the full database of GAO's Open Recommendations »

Sep 30, 2014

Sep 26, 2014

Sep 25, 2014

Sep 19, 2014

Sep 18, 2014

Sep 12, 2014

Sep 10, 2014

Aug 25, 2014

Looking for more? Browse all our products here