Practical Statistics vs. Theoretical Statistics
If something works and it keeps on working but you don’t know exactly why it works, what would you do? Our view is that you should keep doing it. Not everyone agrees with us. The American Association of Public Opinion Research (AAPOR) convened a task force to study online survey panels, and released their report last month (we posted a summary of findings last week). To us, the most jarring statement in the report was this:
“There currently is no generally accepted theoretical basis from which to claim that survey results using samples from nonprobability online panels are projectable to the general population.”
Even with careful statistical weighting based on demographics, known biases, propensity to be online and partake in surveys, and so on, the report concludes that online panels should not be used to estimate population parameters. Why? Not because this method doesn’t work (in many cases it does) but because there is no statistical theory to explain why it works, in contrast to probability sampling, for which there is solid theory explaining why it works.
Their conclusion is particularly surprising because although statistical inference based on probability sampling has a solid theoretical underpinning, in practice pure probability samples are almost never achieved. Nearly always we are faced with low response rates and non-response biases. We work around this in practical ways (not always supported by extensive theory) to weight the data, adjust for biases, understand the sources of biases, add caveats to our findings, and so on. And sure enough, our efforts tend to work, so we keep doing it.
There is substantial data to show that carefully managed non-probability online panels can be used to estimate certain population parameters depending on the nature of the study and how exact those estimates need to be. Someday our academics and social theorists will help us understand why. Here’s our theory: Social bodies are not composed of individual units (the assumption underlying inferential statistical theory) but nodes of networks such that non-random entry points to measure social forces can provide as much information as a random selection of individuals. We’ll leave it to the future theorists to tell us if we’re right.
In the meantime, there is a lot of research for which using non-probability online panels makes good sense (which the AAPOR report acknowledges), so we’ll keep doing it and we’ll keep extending it into new areas even if the theory can’t keep up. And we’ll keep refining our techniques based on experience and practice to make it work better, and we’ll keep thinking about why it works and lending that insight to the work we do for you.
If you need help or expertise designing and executing your research based on probability or nonprobability samples, online panels or phone, let us know. We know our statistical theory. More importantly, we know our practical statistics. We can help you sort through it all to ensure rigorous research and practical results.
—Joe Hopper, Ph.D.
