This is the accessible text file for GAO report number GAO-06-775 
entitled 'Estimating the Undocumented population: A "Grouped Answers" 
Approach to Surveying Foreign-born Respondents' which was released on 
September 29, 2006. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as part 
of a longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

United States Government Accountability Office: 

GAO: 

Report to the Subcommittee on Terrorism, Technology, and Homeland 
Security, Committee on the Judiciary, U.S. Senate: 

September 2006: 

Estimating the Undocumented Population: 

A "Grouped Answers" Approach to Surveying Foreign-Born Respondents: 

GAO-06-775: 

GAO Highlights: 

Highlights of GAO-06-775, a report to the Subcommittee on Terrorism, 
Technology and Homeland Security, Committee on the Judiciary, U.S. 
Senate 

Why GAO Did This Study: 

As greater numbers of foreign-born persons enter, live, and work in the 
United States, policymakers need more information—particularly on the 
undocumented population, its size, characteristics, costs, and 
contributions. This report reviews the ongoing development of a 
potential method for obtaining such information: the “grouped answers” 
approach. In 1998, GAO devised the approach and recommended further 
study. In response, the Census Bureau tested respondent acceptance and 
recently reported results. GAO answers four questions. (1) Is the 
grouped answers approach acceptable for use in a national survey of the 
foreign-born? (2) What further research may be needed? (3) How large a 
survey is needed? (4) Are any ongoing surveys appropriate for inserting 
a grouped answers question series (to avoid the cost of a new survey)? 
For this study, GAO consulted an independent statistician and other 
experts, performed test calculations, obtained documents, and 
interviewed officials and staff at federal agencies. The Census Bureau 
and DHS agreed with the main findings of this report. DHHS agreed that 
the National Survey of Drug Use and Health is not an appropriate survey 
for inserting a grouped answers question series. 

What GAO Found: 

The grouped answers approach is designed to ask foreign-born 
respondents about their immigration status in a personal-interview 
survey. Immigration statuses are grouped in Boxes A, B, and C on two 
different flash cards—with the undocumented status in Box B. 
Respondents are asked to pick the box that includes their current 
status and are told, “If it’s in Box B, we don’t want to know which 
specific category applies to you.” A random half of respondents are 
shown the card on the left of the figure (Card 1), resulting in 
estimates of the percentage of the foreign-born population who are in 
each box of that card. The other half of the respondents are shown the 
card on the right, resulting in corresponding estimates for slightly 
different boxes. (No one sees both cards.) The percentage undocumented 
is estimated by subtraction: The percentage of the foreign-born who are 
in Box B of one card minus the percentage who are in Box A of the other 
card. 

Figure: Immigration Status Cards 1 and 2: 

[See PDF for Image] 

Source: GAO; Corel Draw (flag and suitcase); DHS (resident alien 
cards). (The actual size of each card is 8-1/2" by 11"). 

[End of Figure] 

The grouped answers approach is acceptable to many experts and 
immigrant advocates—with certain conditions, such as (for some 
advocates) private sector data collection. Most respondents tested did 
not object to picking a box. Research is needed to assess issues such 
as whether respondents pick the correct box. A sizable survey—roughly 
6,000 or more respondents—would be needed for 95 percent confidence and 
a margin of error of (plus or minus) 3 percentage points. The ongoing 
surveys that GAO identified are not appropriate for collecting data on 
immigration status. (For example, one survey takes names and Social 
Security numbers, which might affect acceptance of immigration status 
questions.) Whether further research or implementation in a new survey 
would be justified depends on how policymakers weigh the need for such 
information against potential costs and the uncertainties of future 
research. 

What GAO Recommends: 

GAO makes no new recommendations in this report. 

[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-06-775]. 

To view the full product, including the scope and methodology, click on 
the link above. For more information, contact Nancy R. Kingsbury at 
(202) 512-2700 or kingsburyn@gao.gov. 

[end of Section] 

Contents: 

Letter: 
Results in Brief: 
Background: 
Experts Seem to Accept “Grouped Answers” Questions If Fielded by a 
Private Sector Organization: 
Various Tests Are or May Be Needed: 
Some 6,000 Foreign-Born Respondents Are Needed for “Reasonably Precise” 
Estimates of the Undocumented: 
The Most Efficient Field Strategy Does Not Seem Feasible: 
Observations: 
Agency Comments: 

Appendix I: Scope and Methodology: 

Appendix II: Estimating Characteristics, Costs, and Contributions of 
the Undocumented Population: 

Appendix III: A Review of Census Bureau and GAO Reports on the Field 
Test of the Grouped Answer Method: 

Appendix IV: A Brief Examination of Responses Observed while Testing an 
Indirect Method for Obtaining Sensitive Information: 

Appendix V: The Issue of Informed Consent: 

Appendix VI: A Note on Variances and “Mirror Image” Estimates: 

Appendix VII: Comments from the Department of Commerce: 

Appendix VIII: Comments from the Department of Homeland Security: 

Appendix IX: Comments from the Department of Health and Human Services: 

Appendix X: GAO Contact and Staff Acknowledgments: 

Bibliography: 

Tables: 

Table 1: Approximate Number of Foreign-Born Respondents Needed to 
Estimate Percentage Undocumented within 2, 3, or 4 Percentage Points at 
90 Percent Confidence Level, Using Two-Card Grouped Answers Data: 

Table 2: Approximate Number of Foreign-Born Respondents Needed to 
Estimate Percentage Undocumented, within 2, 3, or 4 Percentage Points, 
at 95 Percent Confidence Level, Using Two-Card Grouped Answers Data: 

Table 3: Survey Appropriateness: Whether Surveys Meet Criteria Based on 
the Grouped Answers Design: 

Table 4: Survey Appropriateness: Whether Surveys Meet Table 3 (Design 
Based) Criteria and Additional Criteria Based on Immigrant Advocates’ 
Views: 

Table 5: Experts GAO Consulted on Immigration Issues or Immigration 
Studies: 

Figures: 

Figure 1: Immigration Status Card 1, Grouped Answers: 

Figure 2: Immigration Status Card 2: 

Figure 3: Cards 1 and 2 Compared: 

Figure 4: SIPP Flash Card: 

Figure 5: Training Card 1: 

Figure 6: Training Card 2: 

Figure 7: Immigration Status Card Tested in GSS: 

Abbreviations
ACS: American Community Survey: 
BLS: Bureau of Labor Statistics: 
CASI: Computer Assisted Self Interview: 
CPS: Current Population Survey: 
DHS: Department of Homeland Security: 
GSS: General Social Survey: 
HHS: Department of Health and Human Services: 
INS: Immigration and Naturalization Service: 
NAWS: National Agricultural Workers Survey: 
NCHS: National Center for Health Statistics: 
NHIS: National Health Interview Survey: 
NORC: National Opinion Research Center: 
NRC: National Research Council: 
NSDUH: National Survey on Drug Use and Health: 
NSF: National Science Foundation
OMB: Office of Management and Budget: 
SAMHSA: Substance Abuse and Mental Health Services Administration: 
SIPP: Survey of Income and Program Participation: 

[End of Section] 

September 29, 2006: 

The Honorable Jon Kyl: 
Chairman: 
The Honorable Dianne Feinstein: 
Ranking Minority Member: 
Subcommittee on Terrorism, Technology and Homeland Security: 
Committee on the Judiciary: 
United States Senate: 

As greater numbers of foreign-born persons enter, live, and work in the 
United States, policymakers and the general public increasingly place 
high priority on issues involving immigrants. Because separate 
policies, laws, and programs apply to different immigration statuses, 
valid and reliable information is needed for populations defined by 
immigration status. However, government statistics generally do not 
include such information. 

The information most difficult to obtain concerns the size, 
characteristics, costs, and contributions of the population referred to 
in this report as undocumented or currently undocumented.[Footnote 1] 
Such information is needed because, for example, large numbers of 
undocumented persons arrive each year, and the Census Bureau has 
realized that information on the size of the undocumented population 
would help estimate the size of the total U.S. population, especially 
for years between decennial censuses.[Footnote 2] More generally, 
information about the undocumented population--and about changes in 
that population--can contribute to policy-related planning and 
evaluation efforts. 

As you know, in 1998, we devised an approach to surveying foreign-born 
respondents about their immigration status.[Footnote 3] This self- 
report, personal-interview approach groups answers so that no 
respondent is ever asked whether he, she, or anyone else is 
undocumented. In fact, no individual respondent is ever categorized as 
undocumented. Logically, however, grouped answers data can provide 
indirect estimates of the undocumented population. Generally, grouped 
answers questions on immigration status would be asked as part of a 
larger survey that includes direct questions on demographic 
characteristics and employment and might include questions on school 
attendance, use of medical facilities, and so forth; some surveys also 
ask specific questions that can help estimate taxes paid. Potentially, 
combining the answers to such questions with grouped answers data can 
provide further information on the characteristics, costs, and 
contributions of the undocumented population. 

We reported the first results of preliminary tests of the grouped 
answers approach, primarily with Hispanic farmworkers, in 1998 and 
1999; the majority of the preliminary test interviews were fielded by 
Aguirre International of Burlingame, California.[Footnote 4] We also 
recommended that the Immigration and Naturalization Service (INS) and 
the Census Bureau further develop and test the method. In response, the 
Census Bureau contracted for a test as part of the 2004 General Social 
Survey (GSS), which is fielded by the National Opinion Research Center 
(NORC) at the University of Chicago, with "core funding" provided by a 
grant from the National Science Foundation (NSF).[Footnote 5] The 
Census Bureau's analysis of the 2004 GSS data became available in 2006. 

In this report, we respond to your request that we review the ongoing 
development of the grouped answers approach and related issues. We 
address four questions: (1) Is the grouped answers approach 
"acceptable" for use in a national survey of the foreign-born 
population?[Footnote 6] (2) What kinds of further research are or may 
be needed, based on the results of tests conducted thus far and expert 
opinion? (3) How large a survey is needed to provide "reasonably 
precise" estimates of the undocumented population, using grouped 
answers data? (4) Are there appropriate ongoing surveys in which the 
grouped answers question series might eventually be inserted (thus 
avoiding the costs of fielding a new survey)? 

To answer these questions, we: 

* consulted private sector experts in immigration issues and studies, 
including immigrant advocates, immigration researchers, and 
others;[Footnote 7] 

* consulted an independent statistical expert, Dr. Alan Zaslavsky, and 
other experts in statistics and surveys;[Footnote 8] 

* reanalyzed the data from the 2004 GSS test and subjected both our 
analysis and the Census Bureau's analysis to review by the independent 
statistical expert; 

* performed test calculations, using specific assumptions; and: 

* identified ongoing surveys that might be candidates for piggybacking 
the grouped answers question series, gathered documents on those 
surveys, and met with officials and staff at the federal agencies that 
conduct or sponsor them.[Footnote 9] 

We also met with other relevant federal agencies.[Footnote 10] Appendix 
I describes our methodology and the scope of our work in more detail. 
We conducted our work in accordance with generally accepted government 
auditing standards between July 2005 and September 2006. 

Results in Brief: 

Acceptance of the grouped answers approach appears to be high among 
immigrant advocates and respondents. The advocates we interviewed 
generally accepted the approach--with provisos such as fielding by a 
university or other private sector organization, appropriate data 
protection (including protections against government misuse), and high- 
quality survey procedures. The independent statistician, reviewing the 
Census Bureau's analysis and our reanalysis of the 2004 GSS test of 
respondent acceptance, concluded that the grouped answers approach is 
"generally usable" for surveys interviewing foreign-born respondents in 
their homes.[Footnote 11] 

Based on the results of the GSS test and on consultations and 
interviews with varied experts, further work is or may be needed to: 

* Expand knowledge about respondent acceptance. For example, the 2004 
GSS test did not cover persons who are "linguistically isolated" in the 
sense that no member of their household age 14 or older speaks English 
"very well".[Footnote 12] 

* Test the accuracy of responses or respondents' intent to answer 
accurately.[Footnote 13] To date, no tests of response accuracy, or the 
intent to answer accurately, have been conducted, although a number of 
relevant designs can be identified. 

Thousands of foreign-born respondents would be needed to obtain 
"reasonably precise" grouped answers estimates of the undocumented 
population.[Footnote 14] Our calculations and work with statisticians 
showed that while many factors are involved and it is not possible to 
guarantee a specific level of precision, roughly 6,000 interviews would 
be likely to be sufficient to support estimates of the size of the 
undocumented population and major subgroups within it (especially high- 
risk subgroups, defined by characteristics such as age 18 to 40, 
recently arrived, employed[Footnote 15]). Quantitative estimates are 
also possible; for example, major program costs associated with the 
undocumented population may also be estimated, given appropriate 
program data. 

None of the ongoing, large-scale national surveys we identified appear 
to be appropriate for piggybacking the grouped answers question series. 
One self-report personal interview survey is fielded by a private 
sector organization (under a contract with a Department of Health and 
Human Services (HHS) agency); however, that survey focuses on the use 
of illegal drugs, and we believe that direct questions on drug use 
might heighten the sensitivity of the questions on immigration status. 
We believe other ongoing surveys to be inappropriate; for example, one 
asks other sensitive questions (on HIV status) and takes respondents' 
names and Social Security numbers. Additionally, the Census Bureau 
fields these surveys. 

Whether further research or a new survey would be justified depends on 
issues such as how policymakers weigh the need for such information 
against potential costs. 

We received comments on a draft of this report from the Department of 
Commerce (Census Bureau), the Department of Homeland Security (DHS), 
and the Department of Health and Human Services (DHHS). The Census 
Bureau and DHS generally agreed with the main findings of the report, 
and DHHS agreed that the National Survey of Drug Use and Health would 
not be appropriate for "piggy-backing" the grouped answers question 
series. These agencies also provided other technical comments (see 
appendices VII, VIII, and IX). 

Background: 

Grouped Answers Reduce "Question Threat" and Allow Indirect Estimates 
of the Undocumented: 

Survey questions about sensitive topics carry a "threat" for some 
respondents, because they fear that a truthful answer could result in 
some degree of negative consequence (at a minimum, social disapproval). 
The grouped answers approach is designed to reduce this threat when 
asking about immigration status. 

Three key points about the grouped answers approach are that: 

1. no respondent is ever asked whether he or she, or anyone else, is 
undocumented; 

2. two pieces of information are separately provided by two subsamples 
of respondents (completely different people--no one is shown both 
immigration status cards); and: 

3. taking the two pieces of information together--like two different 
pieces of a puzzle--allows indirect estimation of the undocumented 
population, but no individual respondent (and no piece of data on an 
individual respondent) is ever categorized as undocumented. 

We discuss each point in some detail.[Footnote 16] 

1. No respondent is ever asked whether he or she is in the undocumented 
category. Unlike questions that ask respondents to choose among 
specific answer categories, the grouped answers approach combines 
answer categories in sets or "boxes," as shown in figure 1. 

Figure 1: Immigration Status Card 1, Grouped Answers: 

[See PDF for image] 

Sources: GAO; Corel Draw (flag and suitcase); DHS (resident alien 
cards). (The actual size of the card is 8-1/2" by 11"). 

[End of figure] 

Box B includes the sensitive answer category---currently 
"undocumented"--along with other categories that are 
nonsensitive.[Footnote 17] 

Each respondent is asked to "pick the Box"--Box A, Box B, or Box C--
that contains the specific answer category that applies to him or her. 
Respondents are told, in effect: If the specific category that applies 
to you is in Box B, we don't want to know which one it is, because 
right now we are focusing on Box A categories.[Footnote 18] 

By using the boxes, the interview avoids "zeroing in" on the sensitive 
answer. The specific categories shown in the boxes in figure 1 are 
grouped so that: 

* one would expect many respondents who are here legally, as well as 
those who are undocumented, to choose Box B,[Footnote 19] and: 

* there is virtually no possibility of anyone deducing which specific 
category within Box B applies to any individual respondent. 

2. Two pieces of information are provided separately by two subsamples 
of respondents (no one is shown both immigration status cards). 
Respondents are divided into two subsamples, based on randomization 
procedures or rotation (alternation) procedures conducted outside the 
interview process. (For example, a rotation procedure might specify 
that within an interviewing area, every other household will be 
designated as subsample 1 or subsample 2.) 

This "split sample" procedure has been used routinely for many surveys 
over the years. As applied to the grouped answers approach, the two 
subsamples are shown alternative flash cards. Immigration Status Card 
1, described above, represents one way to group immigration statuses in 
three boxes. A second immigration status flash card (Immigration Status 
Card 2, shown in figure 2) groups the same statuses differently. 

Figure 2: Immigration Status Card 2: 

[See PDF for image] 

Sources: GAO; Corel Draw (flag and suitcase); DHS (resident alien 
cards). (The actual size of the card is 8-1/2" by 11"). 

[End of figure] 

The alternative immigration-status cards can be thought of as "mirror 
images" in that: 

* the two nonsensitive legal statuses in Box A of Card 1 appear in Box 
B of Card 2 and: 

* the two nonsensitive legal statuses in Box B of Card 1 appear in Box 
A of Card 2. 

However, the undocumented status always appears in Box B. 

Interviewers ask survey respondents in subsample 1 about immigration 
status with respect to Card 1. They ask survey respondents in subsample 
2 (completely different persons) about immigration status with respect 
to Card 2. Each respondent is shown one and only one immigration-status 
flash card. There are no highly unusual or complicated interviewing 
procedures.[Footnote 20] 

Because the two subsamples of respondents are drawn randomly or by 
rotation, each subsample represents the foreign-born population and, if 
sufficiently large, can provide "reasonably precise" estimates of the 
percentages of the foreign-born population in the boxes on one of the 
alternative cards. 

Incidentally, a respondent picking a box that does not include the 
sensitive answer--for example, a respondent picking Box A or Box C in 
figure 1--can be asked follow-up questions that pinpoint the specific 
answer category that applies to him or her. Thus, direct information is 
obtained on all legal immigration statuses. The data on some of the 
legal categories can be compared to administrative data to check the 
reasonableness of responses. Additionally, these data provide estimates 
of legal statuses, which are useful when, for example, policymakers 
review legislation on the numbers of foreign-born persons who may be 
admitted to this country under specific legal status programs. 

3. No individual respondent is ever categorized as undocumented, but 
indirect estimates of the undocumented population can be made. Using 
two slightly different pieces of information provided by the two 
different subsamples allows indirect estimation of the size of the 
currently undocumented population--by simple subtraction. 

The only difference between Box B of Card 1 and Box A of Card 2 is the 
inclusion of the currently "undocumented" category in Box B of Card 1. 
Figure 3 shows both cards together for easy comparison. 

Figure 3: Cards 1 and 2 Compared: 

[See PDF for image] 

Sources: GAO; Corel Draw (flag and suitcase); DHS (resident alien 
cards). (The actual size of the card is 8-1/2" by 11"). 

[End of figure] 

Thus, the percentage of the foreign-born population who are currently 
undocumented can be estimated as follows: 

* Start with the percentage of subsample 1 respondents who report that 
they are in Box B of Card 1 (hypothetical figure: 62 percent of 
subsample 1). 

* Subtract from this the percentage of subsample 2 who say they are in 
Box A on Card 2 (hypothetical figure: 33 percent of subsample 2). 

* Observe the difference (29 percent, based on the hypothetical 
figures); this represents an estimate of the percentage of the foreign- 
born population who are undocumented. 

Alternatively, a "mirror-image" estimate could be calculated, using Box 
B of Card 2 and Box A of Card 1.[Footnote 21] 

To estimate the numerical size of the undocumented population, a 
grouped answers estimate of the percentage of the foreign-born who are 
undocumented would be combined with a census figure. For example, the 
2000 census counted 31 million foreign-born, and the Census Bureau 
issued an updated estimate of 35.7 million for 2005. The procedure 
would be to simply multiply the percent undocumented (based on the 
grouped answers data and the subtraction procedure) by a census count 
or an updated estimate for the year in question. 

These procedures ensure that no respondents--and no data on any 
specific respondent--are ever separated out or categorized as 
undocumented, not even during the analytic process of making indirect, 
group-level estimates. 

To further ensure reduction of "question threat," the grouped answers 
question series begins with flash cards that ask about nonsensitive 
topics and familiarize respondents with the 3-box approach. For each 
nonsensitive-topic card, interviewers ask the respondent which box 
applies to him or her, saying: If it's Box B, we do not want to know 
which specific category applies to you. 

In this way, most respondents should understand the grouped answers 
approach before seeing the immigration-status card. 

To help ensure accurate responses, respondents who choose Box A can be 
asked a series of clarifying questions.[Footnote 22] (No follow-up 
questions are addressed to anyone choosing Box B.) The questions for 
Box A respondents are designed to prompt them to, essentially, 
reclassify themselves in Box B, if that is appropriate.[Footnote 23] 

The grouped answers question series can potentially be applied in a 
large-scale general population survey, where the questions on 
immigration status would be added for the foreign-born respondents-- 
provided that an appropriate survey can be identified. If a new survey 
of the general foreign-born population were planned, it would involve 
selecting a general sample of households and then screening out the 
households that do not include one or more foreign-born persons. 

Finally, we note that while the initial version of the grouped answers 
approach involved three alternative flash cards (and was termed the 
"three-card method"), we recently devised the version described here, 
which uses two cards rather than three. The two-card method is simpler, 
is easier to understand, and provides more precise estimates. All cards 
are alike in that they feature three boxes in which specific answer 
categories are grouped. 

Characteristics, Costs, and Contributions Can Potentially Be Estimated: 

Generally, grouped answers questions on immigration status would be 
asked as part of a larger survey that includes direct questions on 
demographic characteristics and employment and might include questions 
on school attendance, use of medical facilities, and so forth; some 
surveys also ask specific questions that can help estimate taxes paid. 
Potentially, combining the answers to such questions with grouped 
answers data can be used to provide further information on the 
characteristics, costs, and contributions of the undocumented 
population. 

For example, the numbers of undocumented persons in major subgroups -- 
such as demographic or employment status subgroups--can be estimated, 
provided that the sample of foreign-born persons interviewed is 
sufficiently large. 

Grouped answers data collected from adult respondents can also be used 
to estimate the number of children in various immigration statuses, 
including undocumented--provided that an additional question is 
asked.[Footnote 24] Additionally, when combined with separate 
quantitative data (for example, data on program costs per individual), 
grouped answers data can be used to estimate quantitative information 
(such as program costs) for the undocumented population as a whole--or, 
again, depending on sample size, for specific subgroups. 

The procedures for deriving these more complex indirect estimates are 
described in appendix II. No grouped answers respondent is ever 
categorized as undocumented. 

Statistical Information Is Needed on the Undocumented Population: 

The foreign-born population of the United States is large and growing-
-as is the undocumented population within it. Congressional 
policymakers, the U.S. Commission on Immigration Reform, and the 
National Research Council's (NRC) Committee on National Statistics have 
indicated a need for statistical information on the undocumented 
population, including its size, characteristics, costs, and 
contributions. 

The Census Bureau estimates that as of 2005, foreign-born residents 
(both legally present and undocumented) numbered 35.7 million and 
accounted for at least one-tenth of all persons residing in each of 15 
states and the District of Columbia.[Footnote 25] These figures 
represent substantial increases over the prior 15 years. For example, 
in 1990 the foreign-born population totaled fewer than 20 million; only 
3 states had a population more than one-tenth foreign-born. One result 
is that as the Department of Labor has testified, foreign-born workers 
now constitute almost 15 percent of the U.S. labor force, and the 
numbers of such workers are growing.[Footnote 26] 

A new paper from the Department of Homeland Security (DHS) puts the 
"unauthorized" immigrant population at 10.5 million as of January 2005 
and indicates that if recent trends continued, the figure for January 
2006 would be 11 million.[Footnote 27] The Pew Hispanic Center's 
indirect estimate of the undocumented population as of 2006 is 11.5 
million to 12 million. These estimates represent roughly one-third of 
the entire foreign-born population.[Footnote 28] DHS has variously 
estimated the size of the undocumented population as of January 2000 as 
7 million and 8.5 million.[Footnote 29] Government and other estimates 
for 1990 numbered only 3.5 million.[Footnote 30] 

These various indirect estimates of the undocumented population are 
based on the "residual method." Residual estimation (1) starts with a 
census count or survey estimate of the number of foreign-born residents 
who have not become U.S. citizens and (2) subtracts out estimated 
numbers of legally present individuals in various categories, based on 
administrative data and assumptions (because censuses and surveys do 
not ask about legal status). The remainder, or residual, represents an 
indirect estimate of the size of the undocumented population. 

To illustrate the role of administrative data and assumptions, residual 
estimates draw on counts of the number of new green cards issued each 
year. But they also require assumptions to account for emigration and 
deaths among those who received green cards in earlier years. 

A recent DHS paper providing residual estimates of the undocumented 
population includes ranges of estimates based on alternative 
assumptions made for two key components.[Footnote 31] For example, "by 
lowering or raising the emigration rates 20 percent . . . the estimated 
unauthorized immigrant population would range from 10.0 million to 11.0 
million."[Footnote 32] The DHS paper also lists assumptions that were 
not subjected to alternative specifications. We believe the DHS paper 
represents an advance because, up to now, analysts producing residual 
estimates have generally not made public statements regarding the 
precision of the estimates. (Some critics have, however, indicated that 
residual estimates are likely to lack precision.[Footnote 33]) 

While the residual approach has been used to profile the undocumented 
population on two characteristics--age and country of birth--it is 
limited with respect to estimating (1) current geographic location and 
(2) current employment and benefit use. The reason is that current 
characteristics of legally present persons are not maintained in 
administrative records; analysts must therefore rely largely on 
assumptions.[Footnote 34] In contrast, the grouped answers method does 
allow for the possibility of estimating current characteristics based 
on current self-reports. 

During the mid-1990s, the U.S. Commission on Immigration Reform 
determined that better statistical "information on legal status and 
type of immigrant [is] crucial" to assessing immigration policy. 
Indeed, the Commission called for a variety of improvements in 
estimates of the costs and benefits associated with undocumented 
immigration.[Footnote 35] NRC's Committee on National Statistics 
further emphasized the need for better information on costs, especially 
state and local costs.[Footnote 36] (If successfully fielded, the 
grouped answers method might help provide general information on such 
costs--and, potentially, specific information for large states such as 
California. Sample size limitations would be likely to prohibit 
separate analyses for specific local areas, small states, and states 
with low percentages of foreign-born or undocumented.) 

Over the years, we have received numerous congressional requests 
related to estimating costs associated with the undocumented 
population.[Footnote 37] Recent Census Bureau research and conferences 
reflect the realization that undocumented immigration is a key 
component of current population growth and that there is a resultant 
need for information on this group.[Footnote 38] Additionally, some of 
the immigrant advocates we interviewed expressed an interest in being 
able to better describe the contributions of the undocumented 
population. 

Surveys Are a Key Information Source: 

Various national surveys ask foreign-born respondents to provide 
information about themselves and, in some cases, other persons in their 
households. While such surveys provide a wealth of information on a 
wide variety of areas, including some sensitive topics, national 
surveys generally do not ask about current immigration status--with the 
exception of a question on U.S. citizenship, which is included in 
several surveys. 

As we reported earlier, it is believed that direct questions on 
immigration status "are very sensitive, and negative reactions to them 
could affect the accuracy of responses to other questions on [a] 
survey."[Footnote 39] Two surveys that have asked respondents directly 
about immigration status for several years are: 

* the National Agricultural Workers Survey (NAWS), an ongoing annual 
cross-sectional self-report survey of farmworkers, fielded by Aguirre 
International, a private sector firm under contract to the Department 
of Labor, since 1988,[Footnote 40] and: 

* the Survey of Income and Program Participation (SIPP), a longitudinal 
panel survey of the general population, conducted by the Census Bureau, 
which has asked immigration status questions since 1996. 

Of the two, SIPP is the more relevant, because its immigration status 
questions have been administered to a sample of the general foreign- 
born population. 

SIPP has asked an adult respondent-informant from each household to 
provide information about himself or herself and about others in his or 
her household, including which immigration-status category applied to 
each person when he or she came to this country. Answers are 
facilitated by a flash card that lists major legal immigration statuses 
(see fig. 4).[Footnote 41] A further question asks whether each person 
obtained a green card after arriving in this country. The SIPP 
questions come close to asking about--but do not actually allow an 
estimate of--the number of foreign-born U.S. residents who are 
currently undocumented.[Footnote 42] According to the Census Bureau, 
SIPP is now scheduled to be "reengineered," but the full outlines of 
the revised effort have not been set. 

Figure 4: SIPP Flash Card: 

[See PDF for image] 

Source: U.S. Bureau of the Census. (The actual size of the card is 8-
12" by 11".) 

[End of figure] 

The Grouped Answers Approach Has Been Tested in Surveys Fielded by 
Private Sector Organizations: 

In the middle to late 1990s, the grouped answers question series was 
subjected to preliminary development and testing with Hispanic 
respondents, including interviews with farmworkers conducted by Aguirre 
International, under contract to GAO.[Footnote 43] In these tests, 
every respondent picked a box.[Footnote 44] However, these interviews 
were not conducted under conditions of a typical large-scale survey in 
which interviewers initiate contact with respondents in their 
homes.[Footnote 45] 

To further test respondents' acceptance of the grouped answers 
approach, the Census Bureau created a question module with 3-box flash 
cards and contracted for it to be added to the 2004 GSS. When 
presenting the survey to respondents, interviewers explained that NORC 
of the University of Chicago fielded the GSS survey, with "core 
funding" from an NSF grant.[Footnote 46] The Census Bureau's question 
module included cards from the three-card version of the grouped 
answers approach--which features only one immigration status category 
in Box A. The cards used were: 

* the two training cards shown in figures 5 and 6[Footnote 47] and: 

* the immigration status card shown in figure 7.[Footnote 48] 

Figure 5: Training Card 1: 

[See PDF for image] 

Sources: GAO; Dominican Republic (illustrations). (The actual size of 
the card is 8-1/2" by 11".) 

[End of figure] 

Figure 6: Training Card 2: 

[See PDF for image] 

Source: GAO. (The actual size of the card is 8-1/2" by 11".) 

[End of figure] 

Figure 7: Immigration Status Card Tested in GSS: 

[See PDF for image] 

Sources: GAO; Corel Draw (flag and suitcase); DHS (resident alien 
cards). (The actual size of the card is 8-1/2" by 11".) 

[End of figure] 

Training card 1 shows different types of houses arranged in three 
boxes. Respondents are asked to indicate the type of house they lived 
in when in their home country--by picking a box. They are told that if 
the answer is in Box B, we don't need to know which specific type 
applies to them, because right now we are focusing on Box A. 

Training card 2 shows different modes of transportation, again arranged 
in three boxes. Respondents are asked to indicate the mode of 
transportation they used the most recent time they traveled from their 
home country to the United States--again by picking a box. They are 
again told that if it's in Box B, we don't need to know which specific 
mode applies. 

Additionally, the GSS-Census Bureau module asked interviewers to (1) 
judge respondents' understanding of the 3-box format, (2) observe 
whether respondents objected or "kept silent for a while" when 
presented with the immigration status card, and (3) record any comments 
that respondents made about the cards. As the Census Bureau has noted, 
the module was a partial test because only one immigration status card 
was tested. 

Data and documentation from this field test became available in late 
2005. A Census Bureau analysis of these data (completed in 2006 and 
reproduced in full in appendix IV), indicates that of 237 foreign-born 
respondents, 216 (roughly 90 percent) chose a box, 4 gave other 
answers, and 17 refused or said "don't know." The Census Bureau took 
this "as an indication that most foreign-born who are asked about their 
migrant status in this format would understand the question, know the 
answer, and answer willingly." 

Further, the Census Bureau paper stated that: 

* the "overwhelming majority of foreign-born respondents" picked a box 
on the immigration status card without--according to interviewers--any 
objection, hesitation, or periods of silence; 

* while some interviewers did not give a judgment or were confused 
about rating respondents' understanding, about 80 percent of 
respondents were coded as understanding and about 10 percent as 
not;[Footnote 49] and: 

* some respondents' comments, written in by interviewers, indicated 
that although the GSS is a "personal interview" survey, telephone 
interviews had been substituted, in some cases, and this meant that 
respondents could not see the cards--making the use of the 3-box format 
difficult. 

The Census Bureau's paper highlighted various limitations of the 2004 
GSS test, including (1) testing only one immigration status card, (2) 
underrepresenting Hispanics, and (3) in some instances interviewing 
over the telephone (instead of in person), so that respondents did not 
see the flash cards.[Footnote 50] 

Experts Seem to Accept "Grouped Answers" Questions If Fielded by a 
Private Sector Organization: 

The acceptability of the grouped answers approach appears to be high, 
when implemented in surveys fielded by a university or private sector 
organization. Many immigration experts, including advocates, accepted 
the grouped answers approach, although some conditioned their 
acceptance on a quality implementation in a survey fielded by a 
university or other private sector organization. An independent 
statistical expert believed that the grouped answers approach would be 
generally usable with survey respondents. 

Keys to Acceptance Are Fielding by a Private Sector Organization, Data 
Protections, and Quality Implementation: 

Some of the researchers and advocates we contacted were extremely 
enthusiastic about the potential for new data. No one objected to 
statistical, policy-relevant information being developed on the size, 
characteristics, costs, and contributions of the undocumented 
population. Overall, the immigration experts we contacted (listed in 
appendix I, table 5) accepted the grouped-answers question approach-- 
although advocates sometimes conditioned their acceptance on, for 
example, the questions being asked in a survey fielded by a university 
or private sector organization--with data protections built in. Many 
also offered suggestions for maximizing cooperation by foreign-born 
respondents or ideas about how advocacy organizations might help. 

Some advocates indicated that a key condition of their support would be 
that (1) the grouped answers question on immigration status be asked by 
a university or private sector organization and (2) identifiable data 
(that is, respondents' answers linked to personal identifiers) be 
maintained by that organization. Two advocate organizations 
specifically stated that they "could not endorse," or implied they 
would not support, the grouped answers approach, assuming the data were 
collected and maintained by, in one case, the Census Bureau and, in the 
other case, the government. Many other immigration experts and 
advocates preferred that grouped answers data on immigration status be 
collected by a university or other reputable private sector 
organization pledged to protect the data. 

The immigration advocates said that private sector fielding of a 
grouped answers survey and protection of such data from nonstatistical 
uses that might harm immigrants were key issues because: 

* Some foreign-born persons are from countries with repressive regimes 
and thus have more fear of (less trust in) government than the typical 
U.S.-born person. 

* Despite current law protecting individual data from disclosure, some 
persons believe that information collected by a government agency such 
as the Census Bureau is routinely shared (or that in some circumstances 
it might be shared) across government agencies. Further, one advocate 
pointed out that the Congress could change the current law, eliminating 
that protection. (Although the grouped answers approach does not 
identify anyone as undocumented, it does provide some information 
regarding each respondent's immigration status.) 

* Extremely large-scale data collections--notably, the American 
Community Survey (ACS)--can yield estimates for areas small enough that 
if the data were publicly available, they could be used for 
nonstatistical, nonpolicy purposes. Some advocates referred to the 
World War II use of census data to identify the areas where specific 
numbers of persons of Japanese origin or descent resided. They also 
pointed out that Census Bureau data on ethnicity--including counts of 
Arab Americans--are publicly available by zip code. (The Census Bureau, 
unlike other government agencies and private sector survey 
organizations, is associated with extremely large-scale data 
collections, and some persons may not fully differentiate Census Bureau 
data collection efforts of different sizes.) 

* Hostility to or lack of trust in the Census Bureau might result in 
potentially lower response rates for foreign-born persons, based on the 
World War II experience of the Japanese or a more recent incident in 
which Census Bureau staff helped a DHS enforcement unit access publicly 
available data on ethnicity by zip code.[Footnote 51],,HS stated that 
it did not use these data and had not requested the information by zip 
code.[Footnote 52] The Census Bureau clarified its position on 
providing help to others requesting publicly available data.[Footnote 
53] 

Various advocates saw the issues listed above as linked to their own 
acceptance, as well as to respondent acceptance, of a survey. Linking 
these issues to respondent acceptance of a survey was, in some cases, 
echoed by other immigration experts we consulted.[Footnote 54] Some 
immigrant advocates and other immigration experts counseled us that if 
there were an increase in enforcement efforts in the interior of the 
United States (as opposed to border-crossing areas), foreign-born 
respondents' acceptance of the grouped answers questions would be 
likely to decrease--at least, if the questions were asked in a survey 
fielded by the government. 

One advocate expressly stated a preference for a grouped answers survey 
with funding by a nongovernment entity, such as a foundation. We 
discussed with a number of immigrant advocates who objected to a 
government-fielded survey the possibility of a survey fielded by a 
private sector organization with government funding. In some cases, we 
specifically referred to one or both of the following surveys, which 
(1) have been conducted for many years without inappropriate data 
disclosures and (2) ask direct sensitive questions: 

* the National Survey on Drug Use and Health (NSDUH), fielded by RTI 
International under a contract from HHS's Substance Abuse and Mental 
Health Services Administration (SAMHSA), and: 

* the National Agricultural Workers Survey (NAWS), fielded by Aguirre 
International, under a contract from the Department of Labor.[Footnote 
55] 

The advocates' response was generally to accept the concept of 
government funding of a university's or private sector survey 
organization's field work, provided that appropriate protections of the 
data were built into the funding agreement. 

GAO's contract with Aguirre International for early testing of the 
grouped answers approach with farmworker respondents specified that 
data on respondents' answers would be "stripped of person-identifiers 
and related information." Additionally, the GSS "core funding" grant 
with NSF and its contractual arrangements with sponsors of question 
modules--such as the grouped-answers question insert contracted for by 
the Census Bureau--do not involve the transfer of any data other than 
publicly available data, stripped of identifiers, and limited so as to 
avoid the possibility of "deductive disclosure" with respect to 
respondent identities or local areas.[Footnote 56] 

Various advocates said that their acceptance was also contingent on 
factors such as: 

1. high-quality data, including coverage of persons who have limited 
English proficiency, with special attempts to reach those who are 
linguistically isolated (that is, members of households in which no one 
14 or older speaks English "very well") and to overcome other potential 
barriers (such as cultural differences); 

2. appropriate presentation of the survey, including an appropriate 
explanation of its purpose and how respondents were selected for 
interview; and: 

3. transparency--that is, keeping the immigrant community informed 
about or involved in the development and progress of the survey. 

One advocate specifically said that her organization's support would be 
contingent on both (1) the development of more information on 
respondent acceptance within the Asian community--particularly among 
Asians who have limited English proficiency or are linguistically 
isolated--and (2) a survey implementation that is planned to adequately 
communicate with Asian respondents, including those who are 
linguistically isolated or have little education.[Footnote 57] Although 
one-fourth of the 2004 GSS test respondents were Asian, the test was 
conducted in English (allowing help from bilingual household members), 
and no other tests have included linguistically isolated 
Asians.[Footnote 58] 

Advocates and Experts Suggest Ways to Maximize Respondent Cooperation 
and Offer Their Assistance: 

Advocates and other experts made several suggestions for maximizing 
respondent cooperation with a survey using the grouped answers question 
series--that is, maximizing response rates for such a survey as well as 
maximizing authentic participation. 

Advocates suggested that the survey (1) avoid taking names or Social 
Security numbers,[Footnote 59] (2) hire interviewers who speak the 
respondents' home-country language, (3) let respondents know why the 
questions are being asked and how their households came to be selected, 
(4) conduct public relations efforts, (5) obtain the support of opinion 
leaders, (6) select a survey group from a well-known and trusted 
university to collect the data, and (7) ask respondents about their 
contributions to the American economy through, for example, working and 
paying taxes. 

Additionally, survey experts suggested: 

* using audio-Computer Assisted Self Interview (audio-CASI),[Footnote 
60] 

* carefully explaining to respondents how anonymity of response is 
protected, and: 

* paying respondents $25 or $30 for participating in the interview. 

Survey experts viewed these elements as key ways of boosting response 
rates or encouraging authentic responses to sensitive questions. For 
example, NAWS, which uses respondent incentives, achieves extremely 
high response rates within cooperating farms--97 percent in 2002, with 
a $20 payment to farmworkers selected. 

Some immigrant advocates also offered suggestions for how their 
organizations or other advocates might help the effort to develop and 
field the grouped answers approach, including: 

1. providing contacts at local organizations to help with arrangements 
for future research, 

2. developing or reviewing Box A follow-up questions, and: 

3. serving on an advisory board with other representatives from 
immigrant communities.[Footnote 61] 

GSS Data and Independent Statistical Consultant Review Show "General 
Usability" of the Grouped Answers Approach: 

As we report above, the Census Bureau's recent analysis of the 2004 GSS 
grouped answers data concluded that the "overwhelming majority of 
foreign-born respondents" picked a box without objection, hesitation, 
or silence. The Census Bureau reported, more specifically, that roughly 
90 percent (216 of 237 respondents) chose a box, 4 gave other answers, 
and 17 refused to answer or said "don't know." 

Our subsequent analysis excluded 19 of the 237 respondents in the 
Census Bureau analysis because: 

˛ 4 were not foreign-born (for example, 1 had been born abroad to 
parents who had, by the time he was born, become naturalized U.S. 
citizens); 

˛ 1 was not classifiable as either foreign-born or not foreign-born 
(because he did not know whether his parents were born in the United 
States); 

˛ 4 others were known to have been interviewed on the telephone, based 
on written-in interviewers' comments recorded in the computer file (for 
example, one wrote that the respondent could not see the cards because 
the interview was on the telephone); and: 

˛ 10 others were subsequently found to have been interviewed on the 
telephone, based on a special GSS hand check of the interview forms for 
respondents who had refused or said "don't know," which was carried out 
in response to our request.[Footnote 62] 

As a result, in our analysis we found that only 6 personally 
interviewed foreign-born GSS respondents refused or said "don't 
know."[Footnote 63] One of the 6 was an 18-year-old Mexican who told 
the interviewer that he did not know whether or not he was a legal 
immigrant. Additionally, we found that the 4 respondents who gave 
"other answers" had provided usable information (for example, one 
called out that he had a student visa) and thus could be recoded into 
an appropriate box. 

After reviewing the two analyses of the GSS test data--the one that the 
Census Bureau performed and the other we performed--Dr. Zaslavsky 
concluded that: 

The test confirms the general usability of the [grouped-answers 
approach] with subjects similar to the target population for its 
potential large-scale use--that is, foreign-born members of the general 
population. Out of about 218 respondents meeting eligibility criteria 
and who were most likely administered the cards in person (possibly 
including a few who had telephone interviews but responded without 
problems), only 9 did not respond by checking one of the 3 boxes. Of 
these, 3 provided verbal information that allowed coding of a box, and 
6 declined to answer the question altogether. Furthermore, several of 
these [6] raised similar difficulties with other 3-box questions on 
nonsensitive topics (type of house where born, mode of transportation 
to enter United States), suggesting that the difficulties with the 
question format were at least in part related to the format and not to 
the particular content of the answers. Thus, indications were that 
there would not be a systematic bias due to respondents whose 
immigration status is more sensitive being unwilling to address the 3- 
box format. 

Dr. Zaslavsky emphasized the importance of minimizing or completely 
avoiding telephone interviews when using the grouped answers approach-
-or, alternatively, providing advance copies of the cards to 
respondents before interviewing over the telephone.[Footnote 64] (Dr. 
Zaslavsky's written review is presented in full in appendix III.) 

Various Tests Are or May Be Needed: 

The findings on respondent acceptance--that is, the GSS test--raised 
some unanswered questions about acceptance that experts said should be 
addressed. Additionally, the experts said that one or more tests of 
response validity are needed to determine whether respondents "pick the 
correct box" versus systematically avoiding Box B. 

Questions for Further Research Were Suggested by the GSS Test: 

The independent reviewer of the GSS analyses (Dr. Zaslavsky) concluded 
that: 

four issues should be addressed in future field tests: 

(a) Equivalent acceptability of all forms of the response card, 

(b) Usability with special populations including those with low 
literacy, the linguistically isolated, and concentrated immigrant 
populations, 

(c) Methods that avoid telephone interviews, or reduce bias and 
nonresponse due to use of the telephone, 

(d) Use of follow-up questions to improve the accuracy of box choices. 

As the independent expert explained with respect to point (b), GSS 
undercoverage of the foreign-born population occurred at least in part 
because interviews were conducted only in English,although household 
members could help respondents with limited English.[Footnote 65] 
Various colleagues and experts we talked with supported points (a) 
through (d). We further note that points (a) and (c) were covered or 
touched on in the Census Bureau's paper reporting its analysis of the 
2004 GSS data. In our discussions with Census Bureau staff, they also 
mentioned that further tests of acceptance should include (d) follow-up 
questions for Box A respondents. 

Additionally, some advocates and an immigration researcher suggested 
improving the cards, which might minimize the potential for "don't 
know" or inaccurate answers. A survey expert suggested using focus 
groups to further explore respondent perceptions of the cards--and to 
potentially improve them.[Footnote 66] 

Earlier testing covered a key portion of the populations (Hispanic 
farmworkers) cited in (b) above, was conducted in Spanish, and included 
Box A follow-up questions as recommended in (d) above.[Footnote 67] In 
those interviews, every respondent picked a box. However, 

1. No language other than Spanish or English has been used in testing; 
thus, as one immigrant advocate pointed out, no testing has focused on 
linguistically isolated Asians (those living in households in which no 
adult member speaks English). 

2. The interviews with Hispanic farmworkers were not conducted under 
typical conditions of a household survey. 

3. Only one immigration status card was tested with Hispanic 
farmworkers and in the GSS. 

Therefore, we agree that the acceptance-testing issues the experts 
raised should be considered in assessing the grouped answers approach. 

Studies Should Test Whether Respondents Pick the Correct Box: 

Several experts told us that tests of respondent accuracy--or at least 
respondents' intent to respond accurately--should be conducted. These 
experts emphasized that grouped answers data would not be useful if 
substantial numbers of respondents were to systematically avoid picking 
Box B (that is, to not pick the box with the undocumented category). 
However, one immigration study expert believed that if a response 
validity study involved lengthy delays, fielding a grouped answers 
survey should proceed in advance of a validity study. 

We agree with the experts' position that tests are needed to determine 
whether respondents systematically avoid Box B (even after Box A follow-
up check questions). Tests of response validity would ideally be 
conducted with the methods of encouraging truthful answers that experts 
mentioned, such as (1) explaining why the survey is being conducted, 
how the respondent was selected, and how the anonymity of answers is 
ensured, and (2) using audio-CASI and, if appropriate, paying 
respondents for participating in the interview. And, as the Census 
Bureau pointed out, such a study should include the full grouped 
answers question series, including follow-up questions, and it should 
test both Card 1 and Card 2. Even if small numbers of respondents were 
to respond inaccurately, it would be helpful to estimate this and 
adjust for any resulting bias. 

We discussed various approaches to conducting validity studies with 
immigration experts, including immigrant advocates, and with agencies 
conducting surveys. In reviewing these approaches, we found that 
response validity tests vary according to whether they are conducted 
before, during, or after a survey is fielded. 

Before a large-scale survey is conducted. The grouped answers question 
series could be asked of a special sample of respondents for whom the 
answers are known, in advance, by study investigators on an individual- 
respondent basis. Such knowledge might be based, for example, on 
information that recent applicants for green cards have submitted to 
DHS.[Footnote 68] "Firewalls" could be used to prevent survey 
information from being given to DHS.We discussed this approach with 
DHS; however, experts criticized a DHS-based validity study on both 
methodological and public relations grounds.[Footnote 69] An 
alternative source of data on individuals' immigration statuses might 
avoid these problems, but no alternative source has yet been 
identified. 

Before or as part of a large-scale survey. In either situation (that 
is, in a presurvey study or as part of a survey), respondents could be 
asked if they would be willing to participate in special validity-test 
activities in return for a payment of, say, $25 or $30 for each 
activity. Later, after interviewing had been completed in a given 
location--not as part of the interview process--a sample of respondents 
who chose Box A (that is, those who claimed to be here legally) could 
be asked to: 

˛ participate in a focus group in which respondents would discuss how 
they felt answering the grouped answers questions when the interviewer 
came to their house and, also, could possibly be asked to fill out a 
"secret ballot" indicating whether they had answered authentically in 
the earlier home interview; 

* give permission for a record check and provide information that could 
subsequently be used in a record check (for example, their name, date 
of birth, and Social Security number) and permission to check these 
data with the Social Security Administration;[Footnote 70] or: 

* show his or her documentation (for example, green card) to a 
documents expert.[Footnote 71] 

These checks would logically be focused on Box A respondents, for most 
of whom such checks would be less threatening. We believe that it is 
reasonable to assume that most respondents who chose Box B picked the 
correct box. Further, because the survey interview states that there 
are no more questions on immigration if the respondent picks Box B, 
pursuing follow-up validity checks might be deemed inappropriate for 
Box B respondents.[Footnote 72] 

After data are collected. With a large-scale survey, it would be 
possible to conduct comparative analyses after the data were collected. 
We provide three examples.[Footnote 73] 

1. Grouped answers estimates of the percentage undocumented could be 
compared for (a) all foreign-born versus (b) high-risk groups, such as 
those who arrived in the United States within the past 5 or 10 years. 
The expectation would be that with valid responses, a higher estimate 
of the percentage undocumented would be obtained for those who arrived 
more recently--because, for example, persons who had arrived recently 
were not here during the amnesty in the late 1980s.[Footnote 74] 

2. Comparisons could be made of (a) Box A estimates of specific legal 
statuses and the approximate dates received--notably, the numbers of 
persons claiming to have received valid green cards in 1990 or more 
recently--with (b) publicly available DHS reports of the numbers of 
green cards issued from 1990 to the survey date.[Footnote 75] 

3. Analysts could compare (a) grouped answers estimates of the number 
undocumented overall to (b) estimates of total undocumented obtained by 
the residual method.[Footnote 76] 

Wherever possible, Card 1 and Card 2 should be tested separately for 
accuracy of response. 

The advantage of conducting a validity study in advance of a survey is 
that if significant problems surface, adjustments in the approach can 
be made. Or if the problems are substantial and cannot be easily 
corrected--and if the anticipated survey were to be fielded mostly or 
only to collect grouped answers data--then that survey could be 
postponed or canceled. However, the results of validity tests conducted 
during or after a survey can be used to interpret the data and, 
potentially, to adjust estimates if it appears that, for example, 5 to 
10 percent of undocumented respondents had erroneously claimed to be in 
Box A of Card 1. As one expert noted, conducting an advance study does 
not preclude conducting a subsequent study during or after the survey. 

Some 6,000 Foreign-Born Respondents Are Needed for "Reasonably Precise" 
Estimates of the Undocumented: 

Although several factors are involved, and it is not possible to 
guarantee a specific level of precision in advance, we estimate that 
roughly 6,000 foreign-born respondents, or more, would be needed for a 
grouped answers survey.[Footnote 77] As we explain below, this is based 
on (1) a precision requirement (that is, a 95 percent confidence 
interval consisting of plus or minus 3 percentage points), (2) 
assumptions about the sampling design of the survey in which the 
questions are asked, and (3) the assumption that approximately 30 
percent of the foreign-born population is currently undocumented. 

An indirect grouped answers estimate of the undocumented population 
generally requires interviews with more foreign-born respondents than a 
corresponding hypothetical direct estimate would--assuming it were 
possible to ask such questions directly in a major national survey. One 
key reason is that the main sample of foreign-born respondents must be 
divided into two subsamples. Half the respondents answer each 
immigration status card. On this basis alone, one would have to double 
the sample size required for a direct estimate based on a question 
asked of all respondents. Further, the estimate of undocumented, which 
is achieved by subtraction, combines two separate estimates, each 
characterized by some degree of uncertainty.[Footnote 78] 

Determining the number of respondents required for a "reasonably 
precise" estimate of the percentage of the foreign-born population who 
are undocumented involves three key factors: 

1. specification of a precision level--that is, choice of a 90 percent 
or 95 percent confidence level and an interval defined by plus or minus 
2, 3, or 4 percentage points; 

2. information on (or assumptions about) the sampling design for the 
main survey and for subsamples 1 and 2; and: 

3. to the extent possible, consideration of the likely distribution of 
the foreign-born population across immigration status categories, 
including the various legal categories and the undocumented 
category.[Footnote 79] 

With respect to the first factor involved in determining sample size, 
some agencies--for example, the Census Bureau and the Bureau of Labor 
Statistics (BLS)--use the 90 percent confidence level. Other agencies 
use the 95 percent level. 

With respect to the second factor, the sampling design of a large- 
scale, nationally representative, personal-interview survey is based on 
probabilistic area sampling rather than simple random sampling of 
individuals. This often reduces the precision of estimates (relative to 
simple random sampling).[Footnote 80] The reason is that persons 
selected for interview are clustered in a limited number of areas or 
neighborhoods (and residents of a particular neighborhood may tend to 
be similar). It is possible that the design for selecting subsamples 1 
and 2 could increase precision; however, it is not possible to predict 
by how much.[Footnote 81] 

With respect to the third factor, existing residual estimates point to 
a fairly even 3-way split between three main categories-- undocumented, 
U.S. citizen, and legal permanent resident. However, there is some 
uncertainty associated with these estimates, the distribution may vary 
across subgroups, and the percentages may change in future.[Footnote 
82] Therefore, a range of distributions is relevant. 

Taking each of these factors into account (to the extent possible) and 
using conservative assumptions, we estimated the approximate numbers of 
respondents required for indirect estimates of the undocumented 
population that are "reasonably precise." 

Table 1 shows required sample sizes for the 90 percent confidence 
level, table 2 for the 95 percent level, with precision at plus or 
minus 2, 3, and 4 percentage points. In estimating these required 
sample sizes, we made conservative assumptions and specified a range of 
possibilities for the distribution with respect to the undocumented 
category. 

To identify a single, rough figure for the sample size needed for 
reasonably precise estimates, we focused on: 

1. the 95 percent level, which is more certain and, we believe, 
preferable; 

2. the 30 percent column, because a current residual estimate of the 
undocumented population is in this range; and: 

3. the middle row (for plus or minus 3 percentage points), which is a 
midpoint within the area of "reasonable precision" as defined above. 

With this focus, we estimate that roughly 6,000 or more respondents 
would be required.[Footnote 83] 

Table 1: Approximate Number of Foreign-Born Respondents Needed to 
Estimate Percentage Undocumented within 2, 3, or 4 Percentage Points at 
90 Percent Confidence Level, Using Two-Card Grouped Answers Data: 

Estimate with 2, 3, or 4 percentage points: 2; 
Percent undocumented foreign-born (range of possibilities): 10%: 
10,700; 
Percent undocumented foreign-born (range of possibilities): 30%: 9,900; 
Percent undocumented foreign-born (range of possibilities): 50%: 8,100; 
Percent undocumented foreign-born (range of possibilities): 70%: 5,500; 
Percent undocumented foreign-born (range of possibilities): 90%: 2,100. 

Estimate with 2, 3, or 4 percentage points: 3; 
Percent undocumented foreign-born (range of possibilities): 10%: 4,800; 
Percent undocumented foreign-born (range of possibilities): 30%: 4,400; 
Percent undocumented foreign-born (range of possibilities): 50%: 3,600; 
Percent undocumented foreign-born (range of possibilities): 70%: 2,500; 
Percent undocumented foreign-born (range of possibilities): 90%: 900. 

Estimate with 2, 3, or 4 percentage points: 4; 
Percent undocumented foreign-born (range of possibilities): 10%: 2,700; 
Percent undocumented foreign-born (range of possibilities): 30%: 2,500; 
Percent undocumented foreign-born (range of possibilities): 50%: 2,000; 
Percent undocumented foreign-born (range of possibilities): 70%: 1,400; 
Percent undocumented foreign-born (range of possibilities): 90%: 500. 

Source: GAO analysis. 

Note: Estimated numbers of respondents were calculated assuming that 
(1) foreign-born persons who are not undocumented are evenly split 
between the legal statuses in Box A, Card 1, and Box A, Card 2 (a 
conservative assumption in that it maximizes the required number of 
respondents), (2) sample selection design for the main survey and for 
subsamples 1 and 2 increases the variance of an estimate of 
undocumented by 1.6 (which does not build in potential reductions in 
variance that might occur with a careful design for the selection of 
subsamples 1 and 2); and (3) for simplicity, no respondents choose Box 
C. 

[End of table] 

Table 2: Approximate Number of Foreign-Born Respondents Needed to 
Estimate Percentage Undocumented, within 2, 3, or 4 Percentage Points, 
at 95 Percent Confidence Level, Using Two-Card Grouped Answers Data: 

Estimate with 2, 3, or 4 percentage points: 2; 
Percent undocumented foreign-born (range of possibilities): 10%: 
15,200; 
Percent undocumented foreign-born (range of possibilities): 30%: 
14,000; 
Percent undocumented foreign-born (range of possibilities): 50%: 
11,500; 
Percent undocumented foreign-born (range of possibilities): 70%: 7,800; 
Percent undocumented foreign-born (range of possibilities): 90%: 2,900. 

Estimate with 2, 3, or 4 percentage points: 3; 
Percent undocumented foreign-born (range of possibilities): 10%: 6,800; 
Percent undocumented foreign-born (range of possibilities): 30%: 
6,200[A]; 
Percent undocumented foreign-born (range of possibilities): 50%: 5,100; 
Percent undocumented foreign-born (range of possibilities): 70%: 3,500; 
Percent undocumented foreign-born (range of possibilities): 90%: 1,300. 

Estimate with 2, 3, or 4 percentage points: 4; 
Percent undocumented foreign-born (range of possibilities): 10%: 3,800; 
Percent undocumented foreign-born (range of possibilities): 30%: 3,500; 
Percent undocumented foreign-born (range of possibilities): 50%: 2,900; 
Percent undocumented foreign-born (range of possibilities): 70%: 2,000; 
Percent undocumented foreign-born (range of possibilities): 90%: 700. 

Source: GAO analysis. 

Note: Estimated numbers of respondents were calculated assuming that 
(1) foreign-born persons who are not undocumented are evenly split 
between the legal statuses in Box A, Card 1, and Box A, Card 2 (a 
conservative assumption in that it maximizes the required number of 
respondents), (2) sample selection design for the main survey and for 
subsamples 1 and 2 increases the variance of an estimate of 
undocumented by 1.6 (which does not build in potential reductions in 
variance that might occur with a careful design for the selection of 
subsamples 1 and 2); and (3) for simplicity, no respondents choose Box 
C. 

[A] This is the approximate number of foreign-born respondents needed 
for an overall estimate of the percentage undocumented with a 
confidence interval of plus or minus 3 percentage points at the 
(preferred) 95% confidence level, assuming that 30% of the foreign-born 
are undocumented. 

[End of Table] 

High-risk subgroups--subgroups with higher percentages of undocumented 
(such as adults 18 to 44 and persons who arrived in the United States 
within the past 10 years)--would require fewer respondents for the same 
level of precision, as illustrated in the tables' middle and right 
columns. For example, if about 70 percent of a subgroup were 
undocumented, a survey with about 3,500 respondents in that subgroup 
would produce an estimate of the percentage of the subgroup that is 
undocumented, correct to within approximately plus or minus 3 
percentage points at the 95 percent confidence level. 

Low precision could obtain for smaller subgroups in which there are 
relatively few undocumented persons (for example, 10 percent or less), 
particularly if--as assumed in tables 1 and 2--there is an even split 
of legally present foreign-born persons across the Box A categories of 
immigration status cards 1 and 2.[Footnote 84] 

The independent statistician we consulted indicated that if more than 
one grouped answers survey is conducted, combining data across two or 
more surveys could help provide larger numbers of respondents for 
subgroup analysis. For example, if a large-scale survey were conducted 
annually, analysts could combine 2 or 3 years of data to obtain more 
precise estimates. (One caveat is that combining data from multiple 
survey years reduces the time-specificity associated with the resulting 
estimate.) 

Finally, we note that to estimate the numerical size of the 
undocumented population, 

* A grouped answers estimate of the percentage of the foreign-born who 
are undocumented would be combined with a census count of the foreign- 
born or an updated estimate. For example, the 2000 census counted 31 
million foreign-born persons, and the Census Bureau later issued an 
updated estimate of 35.7 million for 2005. 

* The specific procedure would be to multiply the percentage 
undocumented (based on the grouped answers data and the subtraction 
procedure) by a census count or an updated estimate of the foreign-born 
population for the year in question. 

The precision of the resulting estimate of the numerical size of the 
undocumented population would be affected by (1) the precision of the 
grouped answers percentage estimate, which is closely related to sample 
size, as described above, and (2) any bias in the census count or 
updated estimate of the foreign-born population.[Footnote 85] The 
precision of the grouped answers percentage is taken into account by 
using a percentage range (for example, the estimate plus or minus 3 
percentage points) when multiplying. Although the amount of bias in a 
census count or updated estimate is unknown, we believe that any such 
bias would have a proportional impact on the calculated numerical 
estimate of the undocumented population.[Footnote 86] 

To illustrate the proportional impact, we assume that a census count 
for total foreign-born is 5 percent too low. Using that count in the 
multiplication process would cause the resulting estimate of the size 
of the undocumented population to be 5 percent lower than it should 
be.[Footnote 87] The situation is analogous for subgroups.[Footnote 88] 

Overall, it seems clear that reasonably precise grouped answers 
estimates of the undocumented population and its characteristics 
require large-scale data collection efforts but not impossibly large 
ones. 

The Most Efficient Field Strategy Does Not Seem Feasible: 

A low-cost field strategy would be to insert the new question series in 
an existing, nationally representative, large-scale survey--that is, to 
pose the grouped answers questions to the foreign-born respondents 
already being interviewed. However, based on our review of on-going 
large-scale surveys, the insertion strategy does not seem feasible. 
Specifically, we identified four potentially relevant surveys but none 
met criteria based on the grouped answers design and other criteria 
based on immigrant advocates' concerns. 

The dollar costs associated with inserting a grouped answers module are 
difficult to calculate in advance because many factors are involved. 
However, to suggest the "ball park" within which the cost of a grouped 
answers insert might be categorized, if an insertion were possible, we 
present the following two examples. 

* The GSS test, in which a grouped answers question module was 
inserted, cost approximately $100 per interview (more than 200 
interviews were conducted). On average, the question series took 3.25 
minutes. Logically, per-interview costs are likely to be higher in 
relatively small surveys than in larger surveys with thousands of 
foreign-born respondents. 

* For the much larger Current Population Survey (CPS), with interviews 
covering native-born and foreign-born persons in more than 50,000 
households, the Census Bureau and BLS told us that "an average 10- 
minute supplement cost $500,000 in 2005."[Footnote 89] This implies $10 
per interview at the 50,000 level, but per-interview costs might be 
higher when the question series applied to only a portion of the 
respondents. Additional costs might apply for flash cards and foreign- 
language interviews. BLS noted that still other costs would apply for 
advance testing and subsequent analyses requested by the customer. 

A more costly option would be to ask the grouped answers question 
series in a follow-back survey of foreign-born respondents identified 
in interviewing for an existing survey. (In-person self-report 
interviews can cost $400 to $600 each.) More costly still would be the 
development of a new, personal-interview survey of a representative 
sample of the foreign-born population devoted to migration issues; the 
main reason is that there would be additional costs in "screening out" 
households without foreign-born persons. 

We identified four potentially relevant ongoing large-scale surveys. 
All have prerequisites and processes for accepting (or not accepting) 
new questions. We also developed six criteria for assessing the 
appropriateness of each survey as a potential vehicle for fielding the 
grouped answers approach. Three criteria are based on design 
requirements, and three are based on the views of immigrant advocates. 
We found that no ongoing large-scale survey met all criteria. 

Four Ongoing Large-Scale Data Collections Sometimes Accept Additional 
Questions: 

We identified four nationally representative, ongoing large-scale 
surveys in which respondents are or could be personally 
interviewed.[Footnote 90] Three of these conduct most or all interviews 
in person: 

1. the Current Population Survey (CPS), sponsored by BLS and the Census 
Bureau and fielded by Census; 

2. the National Health Interview Survey (NHIS), sponsored by the 
National Center for Health Statistics (NCHS) and fielded by the Census 
Bureau; and: 

3. the National Survey on Drug Use and Health (NSDUH), sponsored by 
SAMHSA and fielded by RTI International, a private sector contractor. 

The fourth survey is the American Community Survey (ACS), a much larger 
survey fielded by the Census Bureau and using "mixed mode" data 
collection. The majority of the data are based on mailed questionnaires 
or telephone interviews, with the remaining data based on personal 
interviews. In addition, there is one personal-interview follow-back 
survey that uses the ACS frame and data to draw its sample.[Footnote 
91] Other follow-back surveys might eventually be possible. 

For any of these four surveys, inserting a new question or set of 
questions (or fielding a "follow-back" survey based on respondents' 
answers in the main survey) requires approvals by the Office of 
Management and Budget (OMB), the agencies that sponsor or field the 
surveys, and in cases in which data are collected by a private sector 
organization, the organization's institutional review board. 

The prerequisites for an ongoing survey's accepting new questions 
typically include low anticipated item nonresponse, pretesting and 
pilot testing (including debriefing of respondents and interviewers) 
that indicate a minimum of problems, review by stakeholders to 
determine acceptability, and tests that indicate no effect on either 
survey response rates or answers to the main survey's existing 
questions.[Footnote 92] Another prerequisite would be the expectation 
of response validity.[Footnote 93] 

Additionally, multiple agencies mentioned a need for prior "cognitive 
interviewing," compatibility with existing items (so that there is no 
need to change existing items), and no significant increase in 
"respondent burden" (by, for example, substantially lengthening the 
interview).[Footnote 94] 

Agencies sponsoring or conducting large-scale surveys varied on the 
perceived relevance of immigration to the main topic of their survey. 
For example, BLS noted that some of its customers would be interested 
in data on immigration status by employment status (among the foreign- 
born), and the Census Bureau has indicated the relevance of 
undocumented immigration to population estimation. But some other 
agencies saw little relevance to the large-scale surveys they sponsored 
or conducted. Resistance to including a grouped answers question series 
might occur where an agency perceives little or no benefit to its 
survey or its customers. 

Additionally, one agency raised the issue of informed consent, which we 
discuss in appendix V. 

No Ongoing Large-Scale Data Collection Met Our Criteria: 

Based on the design of the grouped answers approach, as tested to date, 
two criteria for an appropriate survey are (1) personal interviews in 
which respondents can view the 3-box cards and (2) a self-report format 
in which questions ask the respondents about their own status (rather 
than asking one adult member of a household to report information on 
others). A third criterion is that the host survey not include highly 
sensitive direct questions that could affect foreign-born respondents' 
acceptance of the grouped answers questions.[Footnote 95] We based 
these criteria on the results of the GSS test, our knowledge of the 
grouped answers approach, and general logic. 

As shown in table 3, one of the surveys we reviewed (the CPS) does not 
meet the self-report criterion; that is, it accepts proxy responses. 
Two other surveys (the NHIS and NSDUH) do not meet the criterion of an 
absence of highly sensitive questions, since they include questions on 
HIV status (NHIS) and the use of illegal drugs (NSDUH). Conducting a 
follow-back survey based on ACS would meet all three criteria.[Footnote 
96] 

Table 3: Survey Appropriateness: Whether Surveys Meet Criteria Based on 
the Grouped Answers Design: 

Survey type: Ongoing survey; 
Specific survey: Current Population Survey (CPS); 
Three design-based criteria: 1. Are the data gathered in personal 
interviews?: YES> Mostly, for in-person waves; 16% of foreign-born 
interviewed by telephone, in the in-person waves[A]; 
Three design-based criteria: 2. Are all respondents selected to self-
report?: No. An adult respondent reports on self and provides proxy 
responses for others in his or her household. In-person data for 6,744 
households with 1 or more foreign-born members (2006); 
Three design-based criteria: 3. Are direct questions not highly 
sensitive?: YES, not highly sensitive[B]. 

Survey type: Ongoing survey; 
Specific survey: National Health Interview Survey (NHIS); 
Three design-based criteria: 1. Are the data gathered in personal 
interviews?: YES. Mostly; 17% of foreign-born sample adults interviewed 
by telephone; 
Three design-based criteria: 2. Are all respondents selected to self-
report?: YES. For some questions, but not all, 4,829 foreign-born 
adults self-reported (2004); 
Three design-based criteria: 3. Are direct questions not highly 
sensitive?: No. There are direct questions on HIV, other STDs[C]. 

Survey type: Ongoing survey; 
Specific survey: National Survey of Drug Use and Health (NSDUH); 
Three design-based criteria: 1. Are the data gathered in personal 
interviews?: Yes. All interviewed in person; 
Three design-based criteria: 2. Are all respondents selected to self-
report?: Yes. 7,364 foreign-born age 12 and older and 4,934 foreign-
born age 18+ self-reported (2004); 
Three design-based criteria: 3. Are direct questions not highly 
sensitive?: No. There are direct questions on respondent's use and sale 
of drugs like marijuana and cocaine. 

Survey type: Potential follow-back survey; 
Specific survey: Potential American Community Survey (ACS) follow-back 
survey, by the Census Bureau- on all or a sample of foreign-born on 
whom ACS data were collected; 
Three design-based criteria: 1. Are the data gathered in personal 
interviews?: YES. A follow back could specify personal interviews only. 
(ACS is mixed mode, mostly mail); 
Three design-based criteria: 2. Are all respondents selected to self-
report?: YES. A follow-back could specify self-report only. (ACS data 
include both self-report data and proxy data in which one member of a 
household provides responses for others); 
Three design-based criteria: 3. Are direct questions not highly 
sensitive?: Yes, not highly sensitive. 

Source: GAO analysis. 

[A] The CPS includes successive data collections or "waves" to update 
data over time, at selected households. In some waves, interviews are 
conducted in person; in others, by telephone. 

[B] Based on the core CPS questionnaire. (Different modules or 
supplements may be added in particular survey years or CPS waves.) 

[C] HIV refers to human immunodeficiency virus. STDs refers to sexually 
transmitted diseases. 

[End of Table] 

The views of immigrant advocates, which were echoed by some other 
experts, suggested three additional criteria for a candidate "host" 
survey: 

1. data collection by a university or private sector organization, 

2. no request for the respondent's name or Social Security number, and: 

3. protection from possible release of grouped answers survey data for 
small geographic areas (to guard against estimates of the undocumented 
for such areas). 

The experts based their views on (1) methodological grounds (foreign- 
born respondents would be more likely to cooperate, and to respond 
truthfully, if all or some of these criteria were met) and (2) concerns 
about privacy protections at the individual or group levels.[Footnote 
97] These criteria are potentially important, in part because the 
success of a self-report approach hinges on the cooperation of 
individual immigrants and, most likely, also on the support of opinion 
leaders in immigrant communities.[Footnote 98] With respect to the 
first criterion above, we note that with the exception of initial GAO 
pretests, all tests of the grouped answers approach have involved data 
collection by a university or private sector organization. Without 
further tests, we do not know whether acceptance would be equally high 
in a government-fielded survey. 

As shown in table 4, an ACS follow-back would potentially not meet any 
of the three criteria based on immigrant advocates' views. Only one 
survey (NSDUH) met all three criteria based on immigrant advocates' 
views--and because of its sensitive questions on drug use, that survey 
did not meet the design-based table 3 criteria. 

Table 4: Survey Appropriateness: Whether Surveys Meet Table 3 (Design 
Based) Criteria and Additional Criteria Based on Immigrant Advocates' 
Views: 

Survey type: Ongoing survey; 
Specific survey: Current Population Survey (CPS); 
Meets all table 3 (design based) criteria: No; 
Three additional criteria based on immigrant advocates' views: 1. Does 
a nongovernment organization conduct field work?: No. The Census Bureau 
conducts field work[B]; 
Three additional criteria based on immigrant advocates' views: 2. Are 
interviews anonymous (that is, no names or Social Security Numbers are 
taken)?: No. Takes names. 
Three additional criteria based on immigrant advocates' views: 3. Is 
sample too small for reliable small-area estimates of undocumented?[A]: 
YES. 

Survey type: Ongoing survey; 
Specific survey: National Health Interview Survey (NHIS); 
Meets all table 3 (design based) criteria: No; 
Three additional criteria based on immigrant advocates' views: 1. Does 
a nongovernment organization conduct field work?: No. The Census Bureau 
conducts field work[C]; 
Three additional criteria based on immigrant advocates' views: 2. Are 
interviews anonymous (that is, no names or Social Security Numbers are 
taken)?: No. Takes both names and Social Security numbers; 
Three additional criteria based on immigrant advocates' views: 3. Is 
sample too small for reliable small-area estimates of undocumented?[A]: 
Yes. 

Survey type: Ongoing survey; 
Specific survey: National Survey of Drug Use and Health (NSDUH); 
Meets all table 3 (design based) criteria: No; 
Three additional criteria based on immigrant advocates' views: 1. Does 
a nongovernment organization conduct field work?: Yes; 
Three additional criteria based on immigrant advocates' views: 2. Are 
interviews anonymous (that is, no names or Social Security Numbers are 
taken)?: Yes; 
Three additional criteria based on immigrant advocates' views: 3. Is 
sample too small for reliable small-area estimates of undocumented?[A]: 
Yes. 

Survey type: Potential follow-back; 
Specific survey: Potential American Community Survey (ACS) follow-back 
survey y the Census Bureau- on all or a sample of foreign-born on whom 
data were collected; 
Meets all table 3 (design based) criteria: Yes; 
Three additional criteria based on immigrant advocates' views: 1. Does 
a nongovernment organization conduct field work?: No. Only the Census 
Bureau can conduct field work; 
Three additional criteria based on immigrant advocates' views: 2. Are 
interviews anonymous (that is, no names or Social Security Numbers are 
taken)?: No. Takes names in the initial survey, and a follow-back would 
be based on knowing each person's identity; 
Three additional criteria based on immigrant advocates' views: 3. Is 
sample too small for reliable small-area estimates of undocumented?[A]: 
Potentially, no. A follow-back might be extremely large. (Also, small-
area releases are not prohibited by law or policy). 

Source: GAO analysis. 

Note: Table 3 criteria are personal interviews; respondent reports on 
himself or herself; no highly sensitive direct questions. 

[A] For this report, we define "small area" as below the county level. 

[B] For CPS, only the Census Bureau can conduct a follow-back. 

[C] For NHIS, a follow-back by a private sector organization might be 
possible. 

[End of Table] 

In conclusion, we did not find a large-scale survey that would be an 
appropriate vehicle for "piggybacking" the grouped answers question 
series. 

Observations: 

For more than a decade, the Congress has recognized the need to obtain 
reliable information on the immigration status of foreign-born persons 
living in the United States--particularly, information on the 
undocumented population--to inform decisions about changing immigration 
law and policy, evaluate such changes and their effects, and administer 
relevant federal programs. 

Until now, reliable data on the undocumented population have seemed 
impossible to collect. Because of the "question threat" associated with 
directly asking about immigration status, the conventional wisdom was 
that foreign-born respondents in a large-scale national survey would 
not accept such questions--or would not answer them authentically. 

Testing So Far Affirms That the Grouped Answers Approach Is Promising: 

Using the grouped answers approach to ask about immigration status 
seems promising because it reduces question threat and is statistically 
logical. Additionally, this report has established that: 

* The grouped answers approach is acceptable to most foreign-born 
respondents tested (thus far) in surveys fielded by private sector 
organizations; it is also acceptable--with some conditions, such as 
private sector fielding of the survey--to the immigrant advocates and 
other experts we consulted. 

* A variety of research designs are available to help check whether 
respondents choose (or intend to choose) the correct box. 

* The grouped answers approach requires a fairly large number of 
personal interviews with foreign-born persons (we estimate 6,000) to 
achieve reasonably precise indirect estimates of the undocumented 
population overall and within high-risk subgroups. 

However, the most cost-efficient method of fielding a grouped answers 
question series--piggybacking on an existing survey--does not seem 
feasible. Rather, fielding the grouped answers approach would require a 
new survey focused on the foreign-born. This raises two new questions 
about "next steps"--and the answers depend, in large part, on 
policymaker judgments, as described below. 

Two New Questions about "Next Steps" 

Question 1: Are the costs of a new survey justified by information 
needs? DHS stated (in its comments on a draft of this report) that the 
"information on immigration status and the characteristics of those 
immigrants potentially available through this method would be useful 
for evaluating immigration programs and policies." The Census Bureau 
has indicated that information on the undocumented would help estimate 
the total population in intercensal years. And an expert reviewer 
emphasized that a new survey of the foreign-born would be likely to 
help estimate the total population.[Footnote 99] 

Additionally, policymakers might deem a new survey of the foreign-born 
to be desirable for other reasons than obtaining grouped answers data. 
Notably, an immigration expert who reviewed a draft of this report 
pointed out that a survey focused on the foreign-born might provide 
more in-depth, higher-quality data on that population than existing 
surveys that cover both the U.S.-born and foreign born populations. For 
example, more general surveys, such as the ACS and CPS (1) ask a more 
limited set of migration questions than is possible in a survey focused 
on the foreign-born, (2) are not designed with a primary goal of 
maximizing participation by the foreign-born (for example, are not 
conducted by private sector organizations), and (3) as DHS pointed out 
in comments on a draft of this report, may not be designed to cover 
persons who are only temporarily linked to sampled households, because 
such persons may have arrived only recently in the United States and 
are temporarily staying with relatives.[Footnote 100] 

A new survey aimed at obtaining grouped answers data on immigration 
status would require roughly 6,000 (or more) personal, self-report 
interviews with foreign-born adults. Other in-person, self-report 
interviews in large-scale surveys have cost $400 to $600 each. A major 
additional cost would be obtaining a representative sample of foreign- 
born persons; this would likely require a much larger survey of the 
general population in which "mini-interviews" would screen for 
households with one or more foreign-born individuals. 

We did not study the likely costs of such a data collection or options 
for reducing costs. However, survey costs can be estimated (based on, 
for example, the experience of survey organizations), and policymakers 
can, in future, weigh those costs against the information need--keeping 
in mind the results of research on the grouped answers approach, to 
date, and experts' opinions on research needed. 

Question 2: What further tests of the grouped answers method, if any, 
should be conducted before planning and fielding a new survey? On one 
hand, advance testing could: 

* assess response validity (that is, whether respondents pick--or 
intend to pick--the correct box) before committing funds for a survey 
and in time to allow adjustments to the question series; 

* further delineate respondent acceptance and explore the impact on 
acceptance of factors such as government funding--or funding by a 
particular agency--in order to inform decisions about whether or how to 
conduct a survey;[Footnote 101] and: 

* as suggested in DHS's comments on a draft of this report, help 
determine the cost of a full-scale survey.[Footnote 102] 

On the other hand, extensive advance testing would likely delay the 
survey--and may not be needed because: 

* response validity could be assessed--and respondent acceptance could 
be further delineated--concurrently with or subsequent to the survey 
rather than in advance,[Footnote 103] 

* the need for advance testing of response validity would be lessened 
if policymakers see a need for more or better survey data on the 
foreign-born additional to the need for grouped answers data on 
immigration status (see discussion in question 1, above); 

* the value of advance testing would be lessened if changes in 
immigration law and policy occurred between the time of an advance test 
and the main survey, because such changes could affect the context in 
which the survey questions are asked and, hence, change the operant 
levels of acceptance and validity; and: 

* survey costs can be estimated--albeit more roughly--on the basis of 
the experience of survey organizations. 

Given the arguments for and against advance testing, it seems 
appropriate for these to be weighed by policymakers. 

Agency Comments: 

We provided a draft of this report to and received comments from the 
Department of Commerce, the Department of Homeland Security, and the 
Department of Health and Human Services (see appendices VII, VIII, and 
IX, respectively). The Office of Management and Budget provided only 
technical comments, and the Department of Labor did not comment. 

The Census Bureau agreed with the report's discussion of: 

* the grouped answers method, including its strengths and limitations; 

* the Census Bureau-GSS evaluation, including the conclusions of the 
independent consultant (Alan Zaslavsky); and: 

* the need for a "validity study" to determine whether the grouped 
answers method can "generate accurate estimates" of the undocumented 
population. 

The Census Bureau also provided technical comments, which we used to 
clarify the report, as appropriate. 

The Department of Homeland Security stated that the kinds of 
information that the grouped answers approach would provide, if 
successfully implemented, would be useful for evaluating immigration 
programs and policies. DHS further called for pilot testing by GAO to 
assess the reliability of data collection and to help estimate the 
costs of an eventual survey.[Footnote 104] As we indicate in the 
"observations" section of this report, two key decisions for 
policymakers concern: 

* whether to invest in a new survey and: 

* whether substantial testing is required in advance of planning and 
fielding a survey. 

We believe that depending on the answers to these questions, another 
issue--one we cannot address in this report--would concern identifying 
the most appropriate agency for conducting or overseeing (1) tests of 
the grouped answers and (2) an eventual survey of the foreign-born 
population. However, we believe that conducting or overseeing such 
tests or surveys is a management responsibility and, accordingly, is 
not consistent with GAO's role or authorities. DHS made other technical 
comments which we incorporated in the report where 
appropriate.[Footnote 105] 

The Department of Health and Human Services (HHS) agreed that the NSDUH 
would not be an appropriate vehicle for a grouped answers question 
series. Commenting on a draft of this report, HHS said that the report 
should include more information on variance calculations and on "mirror-
image" estimates.[Footnote 106] Therefore, we (1) added a footnote 
illustrating the variance costs of a grouped answers estimate relative 
to a corresponding direct estimate and (2) developed appendix VI, which 
gives the formula for calculating the variance of a grouped answers 
estimate and discusses "mirror image" estimates. 

Additionally, HHS said that interviewers should more accurately 
communicate with respondents when presenting the three-box cards. We 
believe that the text of appendix V on informed consent, based on our 
earlier discussions with privacy experts at the Census Bureau, deals 
with this issue appropriately. As we state in appendix V, it would be 
possible to explain to respondents that "there will be other interviews 
in which other respondents will be asked about some of the Box B 
categories or statuses." Finally, HHS made other, technical comments, 
which we incorporated in the report, as appropriate. 

The Office of Management and Budget provided technical comments. In 
addition, our discussions with OMB prompted us to re-order some of the 
points in the "observations" section of the report. 

The Department of Labor informed us that it had no substantive or 
technical comments on the draft of the report. 

We are sending copies of this report to the Director of the Census 
Bureau, Secretary of Homeland Security, Secretary of Health and Human 
Services, Secretary of Labor, Director of the Office of Management and 
Budget, and to others who are interested. We will also provide copies 
to others on request. In addition, the report will be available at no 
charge on GAO's Web site at [Hyperlink, http://www.gao.gov]. 

If you or your staff have any questions regarding this report, please 
call me at (202) 512-2700. Contact points for our Offices of 
Congressional Relations and Public Affairs may be found on the last 
page of this report. Other key contributors to this assignment were 
Judith A. Droitcour, Assistant Director, Eric M. Larson, and Penny 
Pickett. Statistical support was provided by Sid Schwartz, Mark Ramage, 
and Anna Maria Ortiz. 

Signed by: 

Nancy R. Kingsbury, Managing Director: 
Applied Research and Methods: 

[End of section] 

Appendix I: Scope and Methodology: 

To gain insight into the acceptability of the grouped answers approach, 
we discussed the approach with numerous experts in immigration studies 
and immigration issues, including immigrant advocates. Table 5 lists 
the experts we met with and their organizations. 

Table 5: Experts GAO Consulted on Immigration Issues or Immigration 
Studies: 

Name and Title: Steve A. Camarota, Director of Research; 
Organization: Center for Immigration Studies. 

Name and Title: Robert Deasy, Director, Liaison and Information; 
Crystal Williams, Deputy Director; 
Organization: American Immigration lawyers Association[A]. 

Name and Title: J. Traci Hong, Director of Immigration Program; Terry 
M. Ao, Director of Census and Voting Programs; 
Organization: Asian American Justice Center[A]. 

Name and Title: Guillermina Jasso, Professor of Sociology; 
Organization: New York University. 

Name and Title: Benjamin E. Johnson, Director of Policy, Immigration 
Policy Center; 
Organization: American Immigration Law Foundation[A]. 

Name and Title: John L. (Jack) Martin, Director, Special Projects; 
Julie Kirchner, Deputy Director of Government Relations; 
Organization: Federation for American Immigration Reform. 

Name and Title: Douglas S. Massey, Professor of Sociology and Public 
Affairs; 
Organization: Princeton University. 

Name and Title: Mary Rose Oakar, President; Thomas A. Albert, Director 
of Government Relations; Leila Laoudji, Deputy Director of Legal 
Advocacy; Kareem W. Shora, Director, Legal Department and Policy; 
Organization: American-Arab Anti-Discrimination Committee[A]. 

Name and Title: Demetrios G. Papademetriou, President; 
Organization: Migration policy Institute. 

Name and Title: Jeffrey S. Passel, Senior Research Associate; 
Organization: Pew Hispanic Center. 

Name and Title: Eric Rodriguez, Director, Policy Analysis Center; 
Michele L. Waslin, Director, Immigration Policy Research; 
Organization: National Council of La Raza[A]. 

Name and Title: Helen Hatab Samhan, Executive Director; 
Organization: Arab American Institute Foundation[A]. 

Name and Title: James J. Zogby, President; Rebecca Abou-Chedid, 
Government Relations and Policy Analyst; Nidal M. Ibrahim, Executive 
Director; 
Organization: Arab American Institute[A]. 

Source: GAO. 

Note: Other immigration experts we briefly consulted with by telephone 
or e-mail or in conversations at an immigration conference included 
George Borjas, Professor of Economics and Public Policy, Harvard 
University; Georges Lemaitre, Directorate for Employment, Labour, and 
Social Affairs, Organisation for Economic Co-operation and Development, 
Paris, France; Enrico Marcelli, Assistant Professor of Economics, 
University of Massachusetts at Boston; Randall J. Olson, Director, 
Center for Human Resource Research, The Ohio State University; and 
Michael S. Teitelbaum, Vice President, Alfred P. Sloan Foundation, New 
York. 

[A] Organization advocating for immigrants or expressly dedicated to 
representing their views. We call such organizations immigrant 
advocates, although some may not, for example, lobby for legislation. 

[End of table] 

To ensure that we identified immigration experts from varied 
perspectives, we consulted Demetrios G. Papademetriou, who is among the 
immigration experts listed in table 5, and Michael S. Teitelbaum, Vice 
President of the Alfred J. Sloan Foundation. With respect to immigrant 
advocates, we sought to include advocates who represented (1) 
immigrants in general, without respect to ethnicity; (2) Hispanic 
immigrants, as these are the largest group of foreign-born residents; 
(3) Asian American immigrants, as these are also a large group; and (4) 
Arab American immigrants, as these have been the target of interior 
(that is, nonborder) enforcement efforts in recent years. 

To determine what the 2004 General Social Survey (GSS) test indicated 
about the acceptability of grouped answers questions to foreign-born 
respondents and its "generally usability" in large-scale surveys, we 
obtained the Census Bureau's report of its analysis of those data, and 
we assessed the reliability of the GSS data through a comparison of 
answers to interrelated questions. Then we: 

* submitted the Census Bureau's report of its analysis to Dr. Alan 
Zaslavsky, an independent expert, for review; 

* developed our own analysis of the GSS data and submitted our paper 
describing that analysis to the same expert;[Footnote 107] and: 

* summarized the expert's conclusions and appended his report and the 
Census Bureau's report (reproduced in appendixes III and IV), as well 
summarizing our conclusions.[Footnote 108] 

We used these procedures to ensure independence, given that the GSS 
test was based on our earlier recommendation that the Census Bureau and 
the Department of Homeland Security (DHS) test the grouped answers 
approach.[Footnote 109] 

To describe additional research that might be needed, we outlined the 
grouped answers approach and reviewed the main conclusions of the GSS 
test in meetings with the immigration experts listed in table 5 and 
with private sector statisticians.[Footnote 110] Additionally, we 
discussed the approach with various federal officials and staff at 
agencies responsible for fielding large-scale surveys.[Footnote 111] 

To assess the precision of indirect estimates, we addressed questions 
to Dr. Zaslavsky, developed illustrative tables showing hypothetical 
calculations under specified assumptions, and subjected those tables to 
review. 

To identify and describe candidate surveys for piggybacking the grouped 
answers question series, we set minimum criteria for consideration 
(nationally representative, mainly or only in-person interviews, and 
data on at least 50,000 persons overall, including native-born and 
foreign-born). Then we identified surveys that met those criteria, 
collected documents concerning the surveys, and interviewed officials 
and staff at federal agencies that sponsored or conducted those 
surveys. We also talked with experts in immigration about additional 
key criteria for selecting an appropriate survey. 

The scope of our work had several limitations. We did not attempt to 
collect new data from foreign-born respondents in a survey, focus 
group, or other format. We did not assess census or survey coverage of 
the foreign-born or undocumented populations.[Footnote 112] We did not 
assess nonresponse rates among foreign-born or undocumented persons 
selected for interview. We did not review alternative methods of 
obtaining estimates of the undocumented. 

While we consulted a number of private sector experts and sought to 
include a range of perspectives, other experts may have other views. 
Finally, we do not know to what extent the broad range of persons who 
compose immigrant communities share the views of the immigrant 
advocates we spoke with. 

[End of section] 

Appendix II: Estimating Characteristics, Costs, and Contributions of 
the Undocumented Population: 

Key Characteristics Can Be Estimated: 

Logically, grouped answers data can be used to estimate subgroups of 
the undocumented population, using the following procedures: 

1. isolate survey data for (a) the subsample 1 respondents who are in 
the desired subgroup, based on a demographic or other question asked in 
the survey (for example, if the survey included a question on each 
respondent's employment, data could be isolated for foreign-born who 
are employed), and (b) subsample 2 respondents in that subgroup; 

2. calculate (a) the percentage of the subsample 1 subgroup respondents 
who are in each box of immigration status card 1 and (b) the percentage 
of subsample 2 subgroup respondents who are in each box of immigration 
status card 2; and: 

3. carry out the subtraction procedure (percentage in Box B, Card 1, 
minus percentage in Box A, Card 2), thus estimating the percentage of 
the subgroup who are undocumented. 

The resulting percentage can be multiplied by a census count or an 
updated estimate of the foreign-born persons who are in the subgroup 
(for example, multiply the estimate of the percentage of employed 
foreign-born who are undocumented by the census count or updated 
estimate of the number of employed foreign-born). 

These steps can be repeated to indirectly estimate the size of the 
undocumented population within various subgroups defined by activity, 
demographics, and other characteristics (such as those with or without 
health insurance) that are asked about in the survey. Without an 
extremely large survey, it would be difficult or impossible to derive 
reliable estimates for subgroups with few foreign-born persons or few 
undocumented persons. Ongoing surveys conducted annually have sometimes 
combined 2 or 3 years of data in order to provide more reliable 
estimates of low-prevalence groups; however, there is a loss of time- 
specificity. 

Some Program Costs Can Be Estimated: 

Program cost data are sometimes available on an average per-person 
basis, and surveys sometimes ask about benefit use. In such cases, the 
total costs of a program associated with a certain group can be 
estimated. Program costs associated with the undocumented population 
might be estimated by either (1) multiplying the estimated numbers of 
undocumented persons receiving benefits by average program costs or (2) 
performing the following procedures: 

1. Isolate survey data for all foreign-born subsample 1 respondents who 
said they were in Box B of Card 1 and estimate each individual 
respondent's program cost.[Footnote 113] Then aggregate the individual 
costs to estimate the total program cost (potentially, millions or 
billions of dollars) associated with the population of foreign-born 
persons defined by the group of immigration statuses in Box B, Card 1. 

2. Isolate data for all foreign-born subsample 2 respondents who said 
they were in Box A of Card 2 and, as above, estimate each individual 
respondent's program costs, aggregating these to estimate the total 
program costs associated with the population of foreign-born persons 
defined by the immigration statuses in Box A, Card 2 (again, 
potentially millions or billions of dollars). 

3. Because the only difference between the immigration statuses in Box 
B, Card 1, and Box A, Card 2, is the inclusion of the undocumented 
status in Box B, Card 1, start with the total program cost estimate for 
all Box B, Card 1, respondents and subtract the corresponding cost 
estimate for Box A, Card 2, respondents. 

The result of the subtraction procedure represents an indirect estimate 
of program costs associated with the undocumented population. A more 
precise cost estimate can be obtained by calculating an additional 
"mirror image" cost estimate--this time, starting with costs estimated 
for respondents in Box B of Card 2 and subtracting costs associated 
with respondents in Box A of Card 1. The two "mirror image" estimates 
could then be averaged. 

The key limitations on such procedures are sample size and the 
representation of key subgroups--for example, foreign-born respondents 
residing in small states and local areas. Thus, for example, it is 
possible that state-level costs associated with undocumented persons 
might be estimated with reasonable precision for a large state or city 
with many foreign-born persons and a relatively high percentage of 
undocumented (potentially, California or New York City) but not for 
many smaller states or areas, unless very large samples (or samples 
focused on selected areas of interest) were drawn. Further work could 
explore the ways that complex analyses could be conducted to help 
delineate costs. 

Contributions Might Be Estimated: 

Contributions can be conceptualized as contributions to the economy 
through work or, potentially, through taxes paid. Such contributions 
might be estimated by combining grouped answers data with other survey 
questions to estimate relevant subgroups, such as employed undocumented 
persons. In complex analyses, these data could potentially be combined 
with other data to help estimate taxes paid. 

Logically, Estimates Can Be Made of Undocumented Children: 

Logically, other quantitative estimates might be obtained through 
procedures similar to those outlined above for estimating program 
costs. For example, the numbers of children in various immigration 
statuses might be estimated by asking an adult respondent how many 
foreign-born children (or how many foreign-born school-age children) 
reside in the household and then--using the 3-box card assigned to the 
adult respondent--asking how many of these children are in Box A, Box 
B, and Box C.[Footnote 114] We note that, thus far, testing has not 
asked respondents to report children's immigration status with the 
grouped answers approach. 

Other Estimates May Be Possible: 

If subsamples 1 and 2 are sufficiently large, it might also be possible 
to estimate the portion of the undocumented population represented by: 

* "overstays" who were legally admitted to this country for a specific 
authorized period of time but remained here after that period expired 
(without a timely application for extension of stay or change of 
status)[Footnote 115] and: 

* currently undocumented persons who are applicants for legal status 
and are waiting for DHS to approve (or disapprove) their application. 

To estimate overstays would require a separate question on whether the 
respondent had entered the country on a temporary visa.[Footnote 116] 
To estimate undocumented persons with pending applications would 
require a separate question concerning pending applications for any 
form of legal status (including, for example, applications for U.S. 
citizenship as well as applications for legal permanent resident status 
and other legal statuses). 

The precision of such estimates would depend on factors such as sample 
size, the percentages of foreign-born who came in on temporary visas or 
who have pending applications of some kind, and the numbers of 
undocumented persons within these groups. 

[End of section] 

Appendix III: A Review of Census Bureau and GAO Reports on the Field 
Test of the Grouped Answer Method: 

A Review of Census Bureau and GAO Reports on the Field Test of the 
Grouped Answer Method: 

Alan Zaslavsky: 
Harvard Medical School: 
July 8, 2006: 

A field test of the "Grouped Answer Method" (GAM) for estimating the 
number of undocumented immigrants was conducted by the National Opinion 
Research Center (NORC) in the context of the 2004 General Social Survey 
(GSS). A descriptive report on this test was prepared by the Bureau of 
the Census and a further report by the Government Accountability Office 
(GAO). This is a review of these two documents, focusing on what is 
shown by the analyses and what questions remain to be answered. (The 
Census Bureau report refers to the method as the "Three Card Method" 
(3CM), but in fact the method could be implemented with two or three 
different card forms.) 

Major findings: 

General usability: The test confirms the general usability of the GAM 
with subjects similar to the target population for its potential large- 
scale use, that is, foreign-born members of the general population. Out 
of about 218 respondents meeting eligibility criteria and who were most 
likely administered the cards in person (possibly including a few who 
had telephone interviews but responded without problems), only 9 did 
not respond by checking one of the 3 boxes. Of these, 3 provided 
information, verbal information that allowed coding of a box, and 6 
declined to answer the question altogether. Furthermore, several of 
these raised similar. difficulties with other 3-box questions on 
nonsensitive topics (type of house where born, mode of transportation 
to enter United States), suggesting that the difficulties with the 
question format were at least in part related to the format and not to 
the particular content of the answers. Thus indications were that there 
would not be a systematic bias due to respondents whose immigration 
status is more sensitive being unwilling to address the 3-box format. 

Telephone administration: Of 232 otherwise eligible respondents, 14 
were identified as telephone respondents. Of these, 10 were identified 
because they were followed up in tracking data after failing to provide 
usable information in response to the GAM item. While it is not known 
how many interviews were done by telephone altogether, the number is 
believed to be only a relatively small fraction of the entire survey. 
Thus, item nonresponse was largely a problem of telephone interviewing. 
The higher nonresponse rate for telephone interviewees was not 
surprising given the complexity of the response format (6 categories 
grouped into 3 boxes), the reliance of the item on the visual metaphor 
of boxes, the use of graphics to assist in remembering the categories, 
and the difficulty of comprehending the categories verbally and 
remembering the groupings while answering. In particular, the way in 
which the 3-box method conceals the sensitive responses would be much 
less obvious in a telephone interview. Unfortunately NORC was unable at 
the present time to tell exactly how many telephone interviews were 
administered altogether, so an item nonresponse rate among telephone 
interviews could not be calculated. (NORC plans to disclose individual 
data on mode of interview (telephone versus in-person) by the end of 
2006, which will make possible calculation of item response rates by 
response mode, mail versus telephone.) However, it seems likely for the 
reasons mentioned, as well as from the concentration of problems in 
telephone interviews, that the success rate of the method for telephone 
respondents would be much lower than for in-person respondents. In 
future implementations of this method it would be crucial to address 
this issue, either by (1) attaching the question to a survey that makes 
relatively little use of telephone interviews, or by (2) sending a card 
to the respondent in advance of the interview that could be referred to 
for visual cues for the item. If these solutions were not practical, 
then it might be possible to develop a verbal form of the item adapted 
to telephone use, but this would require some laboratory testing. 

Limitations of this study: 

Single card form: An important limitation of the NORC field test is 
that only one card form was tested. This was very understandable as a 
design limitation in the test since implementation of a multiform 
protocol adds to the complexity of implementation of a study and might 
well be judged to be excessively burdensome for a supplementary item. 
Nonetheless this means that this test cannot answer questions about 
differential rates of nonresponse or procedural difficulties in 
responding to the items. It is also likely that even with multiple 
forms, this test would have been underpowered to answer more refined 
questions about differential rates of nonresponse. With only 9 
nontelephone item nonrespondents, a split sample comparison would have 
had power to detect only the most extreme differences in nonresponse 
rate. However, it is reasonable to generalize about the 
comprehensibility of the items from this test, even with a single form, 
since the modification of rearranging the options in boxes would not be 
expected to affect the usability of the question. 

GSS coverage limitations: GSS coverage had some limitations that made 
the test unrepresentative of the target population of foreign-born. 
Compared to rates estimated from the Current Population Survey, the 
foreign-born are undercovered by the GSS (8.4% in the GSS versus 14.5% 
in the CPS), with particular undercoverage of recent immigrants and 
those from Latin America. The CPS itself likely undercovers recent 
immigrants, particularly the undocumented, so the undercoverage problem 
might be even greater than revealed by comparison to the CPS. Of 
course, by the same token, the CPS and other existing surveys are 
likely to be affected by undercoverage to some extent. Special methods 
might be required to cover concentrations of immigrant population that 
include high rates of undocumented immigrants. The main concern in 
relation to the conclusions of the field test is whether the 
performance of the items, that is their acceptability and 
comprehensibility, would be different either in these special 
populations or with special method used to target these populations. 
Within the GSS test, the problem cases were not notably concentrated 
among recent immigrants or those with more limited English proficiency. 
This suggests that the methods of the GAM did not rely on highly 
culturally specific references or potentially confusing language. 
However, within a community that is largely made up of undocumented 
immigrants, even a "mixed" box might be regarded as more identifying 
and therefore sensitive than in a more heterogeneous community. For 
example in a migrant labor camp in which there are few citizens, 
identifying oneself as "citizen or undocumented immigrant" (as opposed 
to a noncitizen with legal status) might be regarded as tantamount to 
admitting illegal status, while this would not be the case in a general 
population. 

English only: Another concern is the use of English only in the GSS. 
Many of the issues here are similar to those identified in relation to 
undercoverage of recent immigrants in the preceding paragraph. Indeed 
the restriction to English-speaking respondents might explain some of 
the undercoverage of recent immigrants noted above. The additional 
issue raised specifically by English is whether the instructions are 
clear in other languages. It might be expected, however, that because 
the format of the item is largely graphical, it would not be highly 
sensitive to translation. 

Questions for further study: 

Equivalence of acceptability of the alternative response cards: As 
noted above, only one form of the response card was tested in the GSS 
implementation. Future studies should use all (two or three) 
alternative versions of the card, to evaluate whether item nonresponse 
is equivalent for all of the forms, indicating comparable acceptability 
of the forms. 

Effects of nonresponse and incorrect responses on estimates: The effect 
of problems of nonresponse and noncomprehension on the quality of 
estimates from the GAM depends critically on the exact form they take, 
not just on the percentage of responses that are missing or invalid. If 
the group that does not respond to the item is the same regardless of 
which card form is used, then the effect of nonresponse can be 
understood as simple undercoverage of that nonrespondent group. Thus 
within the respondents the analysis proceeds as if with complete data 
and the unknowns only concern the characteristics of the 
nonrespondents, a group whose size is known. The effects of nonresponse 
can be bounded by assuming alternatively that none or all of the 
nonrespondents are undocumented immigrants. These extremes might be 
implausible, especially if qualitative information about the 
nonrespondents (like that collected in the GSS test, or potentially 
relationships of nonresponse to characteristics from larger 
implementations) suggests that the nonrespondents do not generally look 
like undocumented immigrants. Such an argument could be used to develop 
plausible tighter bounds on the fraction of undocumented immigrants 
overall. A simple assumption would be that the nonrespondents have a 
similar fraction of undocumented immigrants to respondents, which would 
allow use of the respondents to make estimates for the entire 
population. 

If nonresponse depends on which card is presented, the analysis of the 
implications is somewhat more complex, since not only the size of the 
nonrespondent group but also its distribution across categories could 
depend on the card. Note that the latter effect would not be evident if 
nonresponse rates overall are the same across cards. For a simple 
example, suppose that 10% of citizens would decline to respond to the 
card that groups citizens with undocumented immigrants, but would 
respond when citizens are ungrouped. Suppose that legally resident 
noncitizens behave similarly. Then the boxes including undocumented 
immigrants would be reduced by 10% with either card, reducing the 
estimate of undocumented immigrants by the same amount even if all the 
undocumented immigrants responded accurately. Many other such scenarios 
could be constructed. Thus it would be useful to study in larger 
samples the factors associated with refusal to respond, particularly to 
investigate whether the reasons given by the respondents seem to be 
associated with the grouping on the card. The evidence from the GSS 
test, however, do not point in the direction of complex nonresponse 
patterns like those hypothesized in this paragraph. 

Finally, similar issues apply with respect to response errors 
(responding but checking the wrong box). A number of possibilities must 
be considered. If a subgroup of legal immigrants systematically report 
the wrong immigration status (for example legal immigrants authorized 
to work in the United States who check the box for citizens) but this 
is unaffected by the grouping of categories, this will have no effect 
on the estimates for the undocumented. This might be the case, for 
example, if some of these respondents are misinformed about their own 
status or confused about the meaning of the categories. However, if 
they systematically avoid the box for the undocumented (checking that 
for citizens or legal noncitizen immigrants as the case may be), this 
will tend toward underestimation of the undocumented. If some 
undocumented immigrants systematically misreport their status, this 
will also create biases in the estimates, especially if they 
systematically avoid the box containing undocumented status. The GSS 
study does not address this issue. 

Effects of mode and mode alternatives: The GSS results support the view 
that the multiple-card items are usable with in-person interviews but 
more problematical with telephone interviews. Some questions of 
interest include the following: 

(1) Can the problems with telephone surveys be remedied by sending a 
response card before the interview? What would the effect of such a 
card be on rates of difficulties in telephone interviews? 

(2) Is there potential for use of mail as a response mode for GAM 
surveys? A mail survey would benefit from the same graphical 
presentation as with the card used in person, but there would be no 
opportunity to explain the question further to respondents who were 
confused by the format. However, if the method were workable in a mail 
survey, it would open up many more potential applications for the 
method. 

(3) Computer-aided self-interview (CASI) allows a respondent to provide 
answers directly to the computer, without letting them be seen by the 
interviewer. CASI has been used to reduce the effect of sensitive items 
by giving the respondent a greater sense of privacy. Might CASI have a 
similar effect with respect to items about immigration status? 

Special populations: non-English speaking (linguistically isolated), 
low literacy, high density of (undocumented) immigrants: Tests should 
be conducted to evaluate the performance of the items in populations 
with these characteristics, each of which was poorly or not at all 
represented in the GSS and might have an effect on ability or 
willingness to complete the item. 

Screening questions: The description of possible citizenship questions 
in the GAO report (page 17-18) suggests the possibility of doing some 
further screening for citizenship to improve the precision of the 
e