How to Succeed with Biased Samples
Leading practitioners of survey research now agree that online polling and other forms of non-probability sampling can work. The key is to give careful attention to the survey protocols, data collection, cleaning, and back-end weighting so that ultimately the sample being analyzed matches the population you want to know about in as many relevant ways as possible.
Evidence for this continues to mount. A fascinating new article now in peer-review by two professors at Columbia University and two researchers at Microsoft suggests that we can push the biases of our samples even further than we have so far assumed.
These researchers decided to use the Xbox gaming platform to test their election polling, which gave them a sample of opinions that were quite far from representative. Still, they were able to generate accurate predictions with the data. Andrew Gellman, one of the authors, summarized it thus:
After adjusting the Xbox responses via multilevel regression and poststratification, we obtain estimates in line with forecasts from leading poll analysts, which were based on aggregating hundreds of traditional polls conducted during the election cycle. We conclude by arguing that non-representative polling shows promise not only for election forecasting, but also for measuring public opinion on a broad range of social, economic and cultural issues.
Most of us using the Internet for surveys work with samples that are much more carefully curated than Xbox. We deliberately pull samples that reflect a range and diversity of opinion and behavior. Even so, there are some who disparage online sampling and surveys, clinging to the long-lost ideal of pure probability sampling.
We believe it is time to put the debate to rest. Gellman notes, “When you do the adjustment carefully, you can do just about as well with a non-probability sample as with a probability sample.” What we need to focus on is further developing the knowledge of how to adjust our samples.
Need help understanding how much bias you can allow and still have your work succeed? Give us a call at (312) 348-6089 and we would be happy to advise.
