Question Order Effects on Political Interest Reports

Download data and study materials from OSF


This experiment was fielded as part of a TESS telephone survey. Data and materials for all the studies included on this survey is available here.


Principal investigator:

Dominic L. Lasorsa

University of Texas at Austin

Email: lasorsa@mail.utexas.edu

Homepage: http://journalism.utexas.edu/faculty/nick-lasorsa


Sample size: 842

Field period: 10/06/2004-01/19/2005

Abstract

Political interest self-judgments declined when preceded by political knowledge questions (knowledge-interest condition). Knowledge difficulty caused respondents to depress interest reports. The effect lessened when buffering the knowledge questions from the interest question were two items evaluating how well elected officials and news media, respectively, keep one informed(knowledge-evaluations-interest). The evaluations served as excuses for knowledge difficulty. Placed before the knowledge items, however, the evaluations worked not as excuses but as additional evidence of difficulty (evaluations-knowledge-interest). To the series of other unfamiliar political objects which follow in the knowledge questions (e.g., Federal Reserve chair, Iraq capital), the evaluation questions introduce two more unfamiliar objects (elected representative, news media). Instead of serving as remote excuses for knowledge difficulty, the evaluation questions serve as more evidence of knowledge difficult.

Hypotheses

Hypothesis 1. Those placed in a position where they can experience difficulty answering political knowledge questions (knowledge-interest-evaluations; evaluations-knowledge-interest; knowledge-evaluations-interest conditions) will report lower political interest than those who do not experience such difficulty (interest-knowledge-evaluations; interest-evaluations-knowledge; evaluations-interest-knowledge conditions).
Hypothesis 2. Those placed in a position where they are offered a ready excuse for difficulty answering political knowledge questions (knowledge-evaluations-interest condition) will report higher political interest than those not given such an excuse for difficulty (knowledge-interest-evaluations; evaluations-knowledge-interest conditions).
Hypothesis 3. Those placed in a position where they are offered a remote excuse for difficulty answering political knowledge questions (evaluations-knowledge-interest condition) will report higher political interest than those not given such an excuse (knowledge-interest-evaluations condition).
Research questions. Is the effect of knowledge difficulty on interest self-judgments strengthened or weakened among different gender, age, income and education groups?

Experimental Manipulations

Six versions of a survey instrument were administered to a random sample of adults. The order in which three sets of questions was asked determined the six experimental conditions: (1)four political knowledge questions, (2)two questions evaluating how well one's news media and elected representative, respectively,keep one informed, and (3)one question reporting political interest.

Outcomes

Political Interest. (“Would you say that you follow what’s going on in government and public affairs most of the time, some of the time, only now and then, or hardly at all?”)

Summary of Results

In the no-knowledge difficulty conditions, political interest self-judgments were higher­82.9% (EIK), 85.7% (IKE), and 84.4% (IEK)­than in the difficulty conditions­70.5% (KIE), 64.3% (EKI), and 76.4% (KEI). While 84.3% of those who did not experience difficulty (N=357) reported interest, only 70.4% of those who experienced difficulty (N=376) reported it. Reports were significantly depressed by difficulty (X2 = 19.92, df = 1, p < .001; phi = .165, p < .001). When a ready excuse was offered for difficulty, interest reports rebounded. While only 67.6% of those without a ready excuse for difficulty reported interest, 76.4% of those with a ready excuse reported it (X2 = 3.708, df = 1, p < .05; phi = .122). While 70.5% of those without an excuse for difficulty reported interest, only 64.3% of those with a remotely available excuse reported it (X2 = 1.178, df = 1, ns; phi = -.069). The evaluation questions were not “remote excuses” for difficulty but, if anything, contributed to difficulty.

Conclusions

A survey experiment tested alternative explanations for a set of question context effects involving items typically asked in public opinion polls and central to political communication research. Priming is known to be responsible for many question order context effects, which allows them to be eliminated by buffering items from each other. This study shows, however, that priming is not the only source of question context effects. Need for consistency is responsible for the basic effect of knowledge difficult depressing interest reports. Discounting of primed information is responsible for the rebounding effect of evaluations serving as excuses for knowledge difficulty. Susceptibility of political interest reports to question order has practical relevance, as well. For example, since interest in political affairs plays a role in electoral involvement, pollsters have included it as a component in the selection of “likely voters” whose choices are counted when projecting election results.

References

Dominic L. Lasorsa, “Effects of the Subjective Experience of Knowledge Difficulty on Self-Judgment of Political Interest,” presented to the Communication Theory & Methodology Division of the Association for Education in Journalism & Mass Communication, San Francisco, 2006.

Dominic L. Lasorsa, "How The Subjective Experience Of Knowledge Difficulty Affects Political Interest Reports," submitted to Journalism & Mass Communication Quarterly.