Order effects
Also known as: Question-order effects, Carryover effects
The sequence of questions or conditions can change the answers people give.
What it means
Order effects are systematic distortions in responses or measured outcomes caused by the sequence in which questions, items, or experimental conditions are presented. In surveys they include question-order effects, where an earlier item primes a concept, sets a comparison standard, or activates norms of consistency or evenhandedness that color later answers, and response-order effects, where the position of options in a list inflates the chosen one (primacy in visual lists, recency in spoken ones). In within-subject experiments they appear as carryover or practice and fatigue effects, where exposure to one condition contaminates performance in the next. Researchers detect and neutralize them through randomization or systematic counterbalancing of order, such as Latin-square designs, and by inserting buffers between related items. They matter because an instrument that is not robust to order can manufacture or mask effects, threatening the validity of polls, questionnaires, and repeated-measures studies.
Examples
Asking people how happy they are with life and then about their dating life yields a different correlation than asking in the reverse order, because the first question reframes the second.
Interviewers who see the strongest candidate first grade everyone after them more harshly. Run the same five people in reverse order and a different person walks away with the offer.
A tasting where everyone sips the cheap bottle first rates the expensive one more generously than a tasting run the other way round. Palates carry over, so the running order becomes part of the result.
First described in Survey methodology; Schuman & Presser (1981).
Where this comes up
- Response Bias in Surveys: A Behavioral Science PerspectiveSurvey responses aren't always accurate. Learn about the most common types of response bias, why they occur, and practi…