Ask Your Most Critical Questions First
Question order in surveys matters a lot more than most of us realize, which makes it critical to assess what you most want to know before your survey respondents are influenced (or, in the language or survey experts, “primed”) by additional questions on the topic.
Here is a superb example from last fall’s political polling in New Jersey. The polling center at Rutgers University correctly predicted that Christie would win for governor and Booker would win the U.S. Senate seat. But their estimates of how much each would win by, were far off. Booker was predicted to win by 22 percentage points, and he won by half that amount. Christie was predicted to win by 36 percentage points, when in fact he won by 22.
The center commissioned a study to figure out what went wrong, and brought in additional polling experts to analyze their methods. The culprit? Question order. The poll asked “who will you vote for?” after asking detailed questions about each candidate. Since most voters had positive views of the winners going into the poll and were unfamiliar with the challengers, the detailed questions served to accentuate those positive views and remind survey respondents of their unfamiliarity with the others. In contrast, other polls asked about voting intentions first. Those polls estimated final outcomes more accurately.
Matt Friedman of The Star-Ledger reports that the polling center will likely “stop asking the head-to-head questions or ask them in separate polls because simply reversing the questions’ order . . . could bias the responses to the other questions.” Quoting the center’s director: “If our primary goal had been to track the races head to head, we really should have just done a separate poll focused only on that.”
Indeed, the biggest lesson we can learn from their mistake is to build surveys carefully around the most critical questions that need to be answered. That’s easier said than done, because there are always so many more ancillary questions that we want answered as well. But the true art of asking questions is as much about specifying and focusing on the most critical research questions as it is designing a survey to answer them.
