Reasons to Avoid Grid-Format Questions

Among the many sources of potential error that can affect surveys are respondents themselves.  They sometimes misinterpret questions, respond in socially acceptable ways, or give “easy” answers in hopes that a more interesting question is just around the corner.

This is not to say they are bad or fraudulent respondents.  Research shows that the vast majority of survey respondents are careful, thoughtful, and truthful in how they answer survey questions.  The problem with respondent error, it turns out, is poor survey design, which may involve biased or  ambiguous questions, tasks that are too complicated or boring, surveys that are too long, and so on.

Recent research shows that grid-style questions that look like this:

or this:

. . . hurt the reliability of answers from respondents.  The evidence for this conclusion comes from one of the leading academic authorities on survey design, Duane Alwin, at Penn State.  Here we quote from a Harvard reviewer who summarized the findings in a recent journal article:

“Of particular note is Alwin’s finding that the widespread survey practice of presenting items in batteries—sets of consecutive questions using the same response format—tends to yield less reliable responses than presenting them alone or in a series of topically related questions with differing response formats. He conjectures that “[s]imilarity of question content and response format may actually distract a respondent from giving full attention to what information is being asked” (p.180). Some might anticipate that similarity in response format would instead heighten reliability, by raising correlations among items in a battery. Alwin’s reliability estimates do not depend on within-occasion correlations between different items,however: the finding suggests that respondents vary across occasions in how they use a battery’s response format.”

In short, even if you have good respondents who are not “straight-lining” through your grids, they may be focusing more on the task of filling out the grid than on a thoughtful response to each item in the grid.  So instead, lay out your questions one at a time, like this:

And when you’re not sure what to do next, namely how to turn all of that now-reliable data into a story that you can really use, give Versta Research a call.

—Joe Hopper, Ph.D.