Pollsters Disagree on Best Sampling Methods
The last presidential election was a stunning win for online polls and a drubbing defeat for telephone polls.
It provoked a loud and contentious debate among survey methodologists about the accuracy of polls and about how (and whether) to revise our methods. So here we are again, four years later with technology having moved faster than ever. Where do we stand today?
Hard to say, yet. In a few months we’ll know whether current U.S. polls can predict elections, which ones, and hopefully why. But in the meantime, there was a minor, instructive dustup this week among some of the top survey methodologists about probability sampling. Probability sampling is that nearly-impossible-to-attain ideal that still forms the cornerstone of most statistical methods used by market researchers, and it has been an issue of fierce debate since the last election.
The dustup was instructive because it touched on all the same issues, but the tone has notably shifted. It seems that most AAPOR members (American Association of Public Opinion Research) are seeing—or at least waiting for—the end of probability sampling. Some noted that even the possibility of such sampling methods and the statistical powers it gave us for so many years are products (and soon-to-be-artifacts) of history.
Here are some of the edited comments, for those who are interested. It is a semi-public form, but we have redacted proper names:
- AAPOR has consistently made statements that the results from NP [Non-Probability] samples should not be used to make population inferences, which is what this article is all about. Shame on [prestigious news organization].
- I am the deputy editor in the polling department at [prestigious news organization]. I am puzzled by your comment. Surely, you have seen that many news outlets report non-probability polls.
- For the time being, these standards generally require that probability-sampling be employed in order for a survey to be part of the news report. We remain hopeful about the future of online non-probability polling and are open to changing our standards when we feel it is the right time to do so.
- The poll results claim a “margin of error” of 2%. If this is not a probability sample, this claim is at best misleading, if not outright fraudulent.
- We need to expand our toolbox of solutions as society changes under us. I would add that, as such, this is not a time for us to hold to rigid orthodoxy but instead one for expansive thinking and experimentation.
- Very few people pick up phones and respond to our surveys these days. The size of non-cooperation and non-coverage are so large that there is legitimate concern that inferences based on design based or model assisted survey are prone to large biases.
- The invention of design-based telephone surveys and its evolution closely reflects the changes in our society—diffusion of telephones to large portion of American households enabled this methodology. If you think even further, I do not think area frame sampling was quite possible without the invention of the automobile in the 20th century. Unlike 100 years ago, we have lots of information about individuals, and powerful computers. We can take advantage of these technologies and define new survey methods just like our predecessors did.
- Do you believe there is known non-zero probability of any likely voter being included in an RDD telephone poll? Do you know the mixture of landline and mobile coverage? Can you account for the 95% of people that do not respond to calls? There is no hard distinction between probability and non-probability polling. All polls are non-probability, but some try harder than other to be probability-based. The sooner everyone accepts that, the sooner the analytics can properly respond to shifting sources of error for different sample frames.
- As to the ideological debate, I’m well indoctrinated on the importance of probability-based sampling and will likely stick with it the rest of my career; but I can appreciate the wisdom of staying very open minded in these dynamic times. At the end of the day, I suspect it is best if we keep pouring our resources into both finding ways to keep the gold-standard of probability-based sampling alive and well while also exploring NP options that, while potentially less valuable in some respects, may still be more than adequate in others.
The failure of so many recent polls, plus the dramatically shifting landscape of telephones, the internet, plummeting response rates, instant and outrageously complex data, etc., all mean that the methods behind polling and our approaches to surveys are changing. And they are changing fast.
At long last, it seems that the direction of thought among many of our AAPOR colleagues is changing as well!
