Respondents Jump to Conclusions
If you have ever been called to participate in a phone survey, you probably know the routine where you hear a question and then jump in with an answer. A good interviewer will remind you that she needs to read the entire question and all the answer options just to be sure that you offered the best response option.
We can’t do this with self-administered online surveys, but there is a way to minimize error associated with respondents jumping to conclusions: Put all clarifying instructions before asking the question, not after. A recent study published in Public Opinion Quarterly documented that if you put instructions before the question, respondents spend more time answering because they are reading the instructions and answering more carefully. Not surprisingly, their answers are more accurate.
Here is an example of a typical question that has clarifying instructions after the question:
The next question is about your retirement savings.
How much did you contribute to your retirement savings in 2012? For the purpose of this question, include any IRA contributions, any pre or post-tax contributions to a workplace retirement account, plus any contributions from your employer, as well as any other savings specifically for retirement.
Here is a better alternative that puts the clarifying instructions first:
The next question is about your retirement savings.
For the purpose of this question, consider any IRA contributions, any pre or post-tax contributions to a workplace retirement account, plus any contributions from your employer, as well as any other savings specifically for retirement. How much did you contribute to your retirement savings in 2012?
I confess that my inclination in writing survey questions is always to get to the point quickly in as few words as possible (always a good thing!) But if I start adding qualifiers and asking for nuance, I remind myself that the respondents are no longer even listening or reading. That’s when it is time to revise.
Need help? We know a lot about how to ask the right questions that generate insightful answers. Give us a call at (312) 348-6089.
