Read Your Questionnaires Out Loud
In our experience, the people who write the best questionnaires are researchers who spend their time immersed in complex survey methods and quantitative analysis but who focused on qualitative research earlier in their careers. Why? Because they are deeply sensitive to the dual objectives of survey design, namely: (1) Elicit data in specific ways and formats that will feed the exact types of analysis and reporting needed, and (2) Make it easy and natural for respondents to provide that data.
That second objective (make it easy and natural for respondents) was nicely highlighted in a discussion last week among several colleagues from the American Association of Public Opinion Research (AAPOR) about questionnaire design. In debating a suggestion for forced choice questions that “avoid acquiescence bias and superficial processing,” by asking (for example), “During the last 30 days, did you eat pizza, or did you not do that?” one astute colleague voiced a concern about “clunkyness” and “tone:”
On first reading the [question] was not just wordy and likely to increase the length of the interview, it felt unnatural. Even worse it felt “pedantic.” I associated it with the way a teacher lectures their students. The student immediately looks for the tricky part. Was there a double negative that I missed? Let me think about it again? What’s the right answer? Uh oh.
Is the tone issue a problem? I think so. I’ve always looked at a questionnaire as a structured conversation. And critical to keeping the respondent engaged must be the feeling that whatever the respondent says must be treated with respect.
The observation from this colleague that surveys must be written as conversations is a wise one that we should all take to heart. And here is one thing you can do to achieve that: Read your surveys out loud. Hearing your own voice articulate questions will highlight where your surveys get clunky and pedantic. It will help you gauge how they flow and “feel” and sound. It will help you revise to make them better.
We know from qualitative research how important it is to establish conversational rapport and openness with study participants. Keep in mind that it is no different with surveys!