Survey Respondents Are Not Strategists

So why are we asking them to do our jobs? I ask this question a lot when we execute surveys for marketing and business strategists. They’re smart people who want us to synthesize lots of data into compelling stories for their business clients. They love to assemble data into dashboards, visualizations, and grids. But they mistakenly believe there is a direct route from a survey respondent’s brain to the output they wish to present.

Here is one recent example. I noticed my partner (he’s not a researcher) taking a survey, answering a whole series of questions that looked something like this:

It was designed as a multiple-item drag-and-drop card sort. He was supposed to think about each aspect of his experience as a customer—how memorable it was, and how satisfied he was—and then locate his experience on this grid with two axes.

Yikes. The strategist who designed this survey was clearly thinking about a business client and wanting a quadrant analysis with areas of strength, areas for improvement, and so on. But for the research process itself, she should have been thinking about her respondents. She may have loved geometry class in high school, but I can assure you that most of her peers did not. It is not an effective way to have most people report on their customer experience.

When writing surveys, never ask survey respondents to answer two questions—or to consider two dimensions—at a time. Ask a series of simple, one-dimensional questions. Then if you love wonky graphs, you can plot all the answers onto a grid yourself.

My partner was having fun with the survey because the drag-and-drop functionality was appealing and because it was a welcome break from the tedious questions he is usually asked to answer (a problem worth writing about another day!) But he was definitely not giving thoughtful consideration to the multi-dimensional task set before him. For two minutes I watched him plop every card in exactly the same spot.

—Joe Hopper, Ph.D.