How Our Brains Untangle Bad Survey Questions

What if a survey asks you a question that implies a yes-no answer (“Have you ever …”) but then offers you answer options that are different from what it implied (“often, sometimes, rarely, never”)? New research tells us that it depends on how the survey is administered. If the survey is online or on paper, you will probably do just fine, easily switching your line of thinking from yes-no to how-often. If the survey is by phone, you are likely to stumble.

The findings from this new research have important implications for how we design surveys, even at Versta Research. I tend to be excessively careful about questions being consistent with answer options. But given that nearly all of our surveys are self-administered online, it is good to know I can be flexible.

Here are two examples of question-answer “mismatches” from federal surveys (usually the gold standard for rigorous surveys) shown by the researchers to illustrate:

 

 

The researchers were involved in two large-scale federal surveys and were able to design test versions to examine the effect of question-answer mismatches. They looked at five types of mismatches, and then looked at “item nonresponse, response time, and response distributions.”

For the self-administered mail survey, they formulated 18 hypotheses about how mismatches would lead to response errors, and just three of the 18 hypotheses were supported. Out of nine mismatched questions, there were no measurable effects on seven.

For the interviewer-administered phone survey, they formulated 13 hypotheses about how mismatches would lead to response errors. Nine of those 13 hypotheses were supported. Out of seven mismatched questions, just one had no measurable effect.

The authors interpret their findings more cautiously than I do, concluding that “mismatches are detrimental and should be avoided.” While I nearly always avoid mismatches, there are times when I realize a mismatched question is going to work better — maybe it is clearer, more straightforward, less wordy, or it avoids multiple clauses.

This new research gives me confidence that for most of the work we do, respondents can successfully answer such questions. Indeed, the lesson for me is that when it comes to online surveys, the clarity of a question and answer options should always outweigh strict rules against mismatches.

—Joe Hopper, Ph.D.