When to Use (and Avoid) “Select-All” Questions

Survey questions that have multiple response options with an instruction to “select all that apply” are common in our industry, and for good reasons. They are efficient and easy for respondents to answer. They are efficient to lay out graphically. They shorten the length of a survey in terms of both time and space.

The problem with this question format is that it usually underreports the truth. Here is an example to illustrate. Present a list of 12 ice cream flavors and ask respondents to check off all the ones they like. The percentage who select peach will almost certainly be lower than if you asked directly, “Do you like peach ice cream?”

The reasons for this are mostly obvious. People do not go through lists and consider each item carefully, one at a time. They scan for answers that jump out. They look for response options that best answer the question. Finding three or four favorites from a list of 12 seems pretty good, and they move on to the next question.

All of this is well known among experienced researchers, and this week the Pew Research Center issued a healthy reminder based on data from their own internal experiments. Their advice is to avoid select-all questions, instead asking forced-choice (yes/no) question for each item on the list.

But we disagree with their advice, because it does not take into account some important ways in which select-all questions are effective.

First, select-all questions do a good job measuring the overall rank order of selected items. If you want to know the top three ice cream flavors among your respondents, either method of asking will yield the same result. Second, select-all questions effectively measure true favorites or top-of-mind answers. They counteract acquiescence biases that lead survey respondents to say “yes” to everything.

So instead of tossing out all select-all questions, here is what we recommend:

1. If you need an accurate point estimate for a specific item(s) on the list, ask about it separately with a fixed choice (yes/no) question, as Pew recommends.
2. If point estimates are less important and all you need is rank order data, or “most important” items, or top-of-mind favorites, use select-all lists instead.

Everybody loves select-all lists for good reasons: they are easier and more efficient. (Image having to answer 12 questions in a row: Do you like peach ice cream? Do you like chocolate ice cream? Do you like pistachio ice cream? … It would be maddening!) So please continue using them, but use them judiciously depending on the insights you need.

—Joe Hopper, Ph.D.

OTHER ARTICLES ON THIS TOPIC:

When to Use Multi-Check vs. Yes-No Questions