When to Use Multi-Check vs. Yes-No Questions
Here are two ways you might ask a question to document multiple behaviors, purchases, interests, etc. They seem like they would be equivalent, but they are not.
The first is called a multi-check format:
The second is called a yes-no grid:
If respondents were super careful, thoughtful, and unbiased in how they answer, the information you get should be the same. All people (and only people) selecting “Cappuccino” in the first format should select “yes” to Cappuccino in the second format. But they don’t. There are people who select “yes” to Cappuccino in the second format, who will not select “Cappuccino” in the first.
What’s going on here? There are two hypotheses. One is called the acquiescence hypothesis. People prefer to answer surveys in positive, agreeable ways—so maybe they gravitate towards the “yes” column and over-report buying Cappuccino. The other is called the salience hypothesis. When presented with a list of items, people overlook some items unless forced to consider each of them carefully, one at a time.
Recent evidence presented at the annual AAPOR Conference suggests that the salience hypothesis is probably correct, which has important implications for which format you should choose:
- Choose a yes-no format if … it is critical that respondents not overlook an item you need to measure. If you dislike grids, then ask about each item of interest in a series of separate yes-no or similar-type questions. Either way, it forces the person taking your survey to think about, and offer, an explicit response to the question you have.
- Choose a multi-check format if … your need to measure is less precise (often it is) and you are focused on avoiding the tedium and time-suck of grids. The multiple response format is much easier and faster for people to do, which makes it more likely they happily complete your survey instead of feeling annoyed and breaking off before finishing.
At Versta Research we use both types of questions all the time. The key to choosing is to consider ahead of time how we will be using the data and the trade-offs inherent in asking one way vs. another.
