Who You Are Missing with Online Surveys
Nine out of ten Americans (89%) regularly use the Internet. So the old argument that online surveys will miss big chunks of the U.S. population is no longer valid. But an important question remains: Are the remaining one in ten (11%) who are beyond reach with online surveys substantially different from the others? And if so, does that bias the results of online surveys?
The answer is yes, they are different. But no, for the vast majority of questions you might want to ask on a survey, leaving them out will not bias the results. These are the findings from new data released last fall by the Pew Research Center, a social science think tank dedicated to documenting trends in U.S. social and economic life.
Pew looked at hundreds of survey questions from multiple surveys administered to its American Trends Panel. This is a nationally representative panel of U.S. adults living in households, recruited by phone using probability-based sampling methods. One in five ATP panelists (18%) participate in the surveys via mail (including 11% who do not have Internet access) while the remaining participate online.
Here is what they found:
“There is little or no difference between the full (Web and mail combined) sample and Web-only sample in a majority of topics that were asked about. Only nine survey items we tested yielded a difference of 5 or more percentage points, while the other 397 (98%) of the survey items yielded a difference of 4 percentage points or less. More specifically, about two-thirds of the 406 items yielded estimates with a difference of zero or 1 point.”
Not surprisingly, the nine questions that showed substantial bias asked about smartphones, computers, and Internet usage.
Keep in mind one caution, however. The 11% who do not have Internet access definitely skew older than others. On average they have lower incomes, and lower educational attainment. They are also disproportionately African American. So when slicing data by demographic subgroups (and for some survey topics, but not all), Pew researchers found less consistency when comparing the Web+ mail sample vs. the Web-only sample.
I am tempted to conclude by saying “stop worrying” about your online samples. But of course you should always be worrying about whether your samples are biased. This new research is strong evidence that we can overcome many of those biases, and that coverage bias is not a problem for most online surveys. But as always, ensuring unbiased outcomes always takes vigilance and deep involvement in the logistics of your research.
