Does This Phone Survey Flatter Me?
The evidence continues to mount that phone surveys should be a method of last resort. Pew Research Center has released results of an experiment conducted in 2014 to test mode differences in how surveys are administered. The research question was whether people answer survey questions differently when responding to a real person on a telephone vs. how they respond to identical questions on a web survey.
The answer is yes, they respond differently. The research found that mode differences are common and measurable, even if not very large. Most importantly, there is a systematic “social desirability” bias in how people answer questions on the phone. People want to look good when confronted with a real person. As such, in this experimental survey they were more likely to:
- Give higher satisfaction ratings on their family life
- Give higher satisfaction ratings on their social life
- Rate their communities as being excellent places to live
- Report talking with their neighbors
- Rate their health as being excellent
- Acknowledge discrimination among minorities
… and they were less likely to:
- Express extreme negative views of politicians
- Report being unable to afford food
- Report needing recent medical care
- Report not attending religious services
The study concludes:
The fact that telephone respondents consistently exhibit more socially desirable reporting is consistent with a large body of preexisting literature on the topic. For most of these and other differences described here, there is no way to determine whether the telephone or the Web responses are more accurate, though previous research examining questions for which the true value is known have found that self-administered surveys generally elicit more accurate information than interviewer-administered surveys.
There are other reasons to avoid phone surveys, as well. They are extremely expensive. They are risky, unless you use manual telephone dialing. And they don’t work so well. In the last presidential election they did far worse in predicting the outcome than online surveys.
Now add to this list of reasons the problem of social desirability bias. Survey respondents want to offer the most flattering portrait of themselves they can manage. But as a researcher probing for the best insights I can find, I want the whole truth no matter how ugly.