Download data and study materials from OSF
This experiment was fielded as part of a TESS telephone survey. Data and materials for all the studies included on this survey is available here.
Principal investigator:
Alexander L Janus
University of Oxford
Email: alexander.janus@sociology.ox.ac.uk
Homepage: http://www.alexjanus.com/
Sample size: 1320
Field period: 10/13/2005-2/1/2006
I employ a little used yet highly innovative unobtrusive questioning technique known as the list experiment. This technique has two advantages compared to other strategies used by survey researchers to measure sensitive or controversial attitudes: it is easy to implement as part of a large-scale telephone survey, and it is able to guarantee the complete anonymity of respondents’ answers. I use the list experiment to gauge Americans’ attitudes toward “cutting off immigration to the United States” and “same-sex marriages.” I also administer comparable direct questions. I find that a direct question dramatically underestimates support for immigration restrictionism, especially among Democrats, women, college graduates, and people who claim to feel warmly toward Hispanics or Asian Americans. For the most part, I do not find statistically significant differences between the list experiment’s and a direct question’s estimates of support for “same-sex marriages.”
Hypothesis 1: I expect that some Americans, in response to a direct survey question, will conceal support for “cutting off immigration to the United States.”
Hypothesis 2: I expect that some Americans, in response to a direct survey question, will conceal opposition to “same-sex marriages.”
The list experiment is administered as follows. Respondents are randomly assigned to three groups. The interviewer reads all groups these instructions: “Now I am going to read you three/four things that sometimes people oppose or are against. After I read all three/four, just tell me HOW MANY of them you oppose. I don’t want to know which ones, just HOW MANY.” All groups are then given the same three nonsensitive items to choose from: • the federal government increasing assistance to the poor; • professional athletes making millions of dollars per year; • large corporations polluting the environment. One of the groups serves as the baseline, while respondents in the two treatment groups receive an additional sensitive item. For one treatment group the sensitive item is: • cutting off immigration to the United States. The additional item for the other treatment group is: • same-sex marriages. The difference in the mean number of items chosen between a treatment group and the baseline group is attributable to two things: the additional sensitive item and sampling error. Subtracting the means and multiplying by 100 provides an unobtrusive estimate of the percentage of respondents opposed to the sensitive item. The list experiment ensures the anonymity of respondents’ answers because at no time is the respondent required to reveal to the interviewer which specific items she or he opposes, just the number of items.
1. Unobtrusive estimate (produced by the list experiment) of support for “cutting off immigration to the United States.”
2.Obtrusive estimate (produced by a direct question) of support for “cutting off immigration to the United States.”
3. Unobtrusive estimate (produced by the list experiment) of support for “same-sex marriages.”
4.Obtrusive estimate (produced by a direct question) of support for “same-sex marriages.”
For the population as a whole, there was a large and statistically significant (p < 0.01) difference between the unobtrusive and obtrusive estimates of support for “cutting off immigration to the United States.” Forty percent of Americans profess to support cutting off immigration in response to a direct question. The list experiment, on the other hand, estimates that an even more astonishing 62 percent of Americans support such a policy. The difference between the unobtrusive and obtrusive estimates of Americans supporting “same-sex marriages” failed to reach conventional levels of statistical significance (p = 0.14). See the below table for unobtrusive and obtrusive estimates of immigration restrictionism by various characteristics of the respondents.
Across the social sciences investigators query respondents on sensitive issues. The most significant obstacle that investigators face in studies on sensitive attitudes or behavior is that some respondents might conceal unpopular views to appear in a favorable light to the interviewer. The most common strategy survey researchers use to guard against social desirability pressures is an introductory script that assures respondents anonymity and stresses the importance of honest answers. Most investigators also train interviewers to respond to respondents’ answers in a neutral fashion and to carefully manage any nonverbal cues (Phillips and Clancy 1972). The results from this study demonstrate, however, that the most commonly employed strategies are not enough. Only when the complete anonymity of respondents’ answers was guaranteed by the list experiment were respondents forthcoming in expressing their attitudes on immigration.
Janus, Alexander L. 2007. “The List Experiment as an Unobtrusive Measure of Attitudes Toward Immigration.” [Paper presented at the Social Science Quantitative Methods Working Group at the University of California, Berkeley.]
Other reference:
Phillips, Derek L. and Kevin J. Clancy. 1972. "Some Effects of 'Social Desirability' in Survey Studies." The American Journal of Sociology 77:921-940
Janus, Alexander L. 2010. "The Influence of Social Desirability Pressures on Expressed
Immigration Attitudes." Social Science Quarterly 91:928-46.