Consumers Eagerly Answer What You Don’t Ask
Most people who take surveys want to share their opinions, which is important for researchers hoping to get a few nuggets of data from willing respondents. The trouble is, if a survey it not written carefully, a respondent’s urgent desire to share their feelings may bias their answers to other questions.
Two marketing professors at Northwestern University’s Kellogg School of Management recently published a paper in the AMA’s Journal of Marketing Research identifying and documenting this unique kind of bias. They call it “response substitution.”
Suppose you eat at a fancy restaurant, pay a lot of money, have delicious food, but suffer horrid and neglectful service. In your eagerness to rant about your bad experience, you are likely to give poor ratings to the quality of food. Poor chef! He thinks his food is terrible. The waiter did a lousy job, so everyone suffers. But wait. It turns out that if you are told before taking the customer satisfaction survey that you will have an opportunity to share all comments and opinions at the end of the survey, you will rate the food higher, knowing you can vent about the waiter later on.
“Response substitution” is an important source of bias, with an easy solution. The lessons for researchers are this:
1. Know when you are likely to encounter response substitution bias. Situations where people are likely to have strong opinions they want to share (satisfaction with personal experiences, politically polarizing issues, etc.) are where you should take measures to reduce response substitution bias.
2. Protect against it with a short introduction to respondents. Tell them up front that their opinions are valued and they will be able to share all opinions in the survey. Remind them to focus on the specific issues being asked about. Be sure that you do give them an opportunity to answer unasked questions in open-ended format somewhere in the survey.
—Joe Hopper, Ph.D.
