More Research on Phone vs. Online Surveys
Another article was just published in the Spring 2010 issue of Public Opinion Quarterly exploring data quality differences between online surveys and phone surveys. The findings were based on a lab experiment in which subjects completed survey questions either on a computer or over an intercom system with an interviewer. Doing the study in a laboratory isolated the mode effect of computerized self-administered data collection vs. an interview conducted by a human.
The article provides evidence that online surveys offer better data quality compared to telephone surveys and other modes of “oral” interviewing. Data from the computer surveys were superior in terms of:
- Validity. The researchers used two different measures of the same thing, and found higher consistency between the two measures for respondents taking the computer-based survey. This is known as concurrent validity.
- Satisficing. Respondents taking the computer-based survey worked harder to provide accurate answers, which showed up in more variation and differentiation among their answers, and fewer response-order effects.
- Social desirability bias. Those taking computer surveys were more likely to share their true feelings about a controversial topic rather than reporting feelings that would make them look more sympathetic or politically correct.
The findings reinforce those reported in another POQ article published at the end of last year, which outlined the relative strengths and weaknesses of phone versus online surveys. They will also no doubt feed the ongoing controversy about usage of online survey panels (See “How Good Are Online Survey Panels?” and “Practical Statistics vs. Theoretical Statistics”).
Which should you do, a phone survey or an online survey? It depends on your objectives. We would be happy to help you think through your options and the strengths of each. Even if there is no “right answer” there is probably a “best answer” within the context of your key objectives.
—Joe Hopper, Ph.D.
