Your Margin of Error Is Bigger than You Think
If you conduct a well-designed, well-executed survey of 500 people you probably think (and yes, we may have told you) that your margin of error is ±4%. It’s beautiful. It’s all based on math. And if you ever had that “aha” moment in statistics class, you probably agree that the mathematical proof of it is beautiful, elegant, and seductive.
But in the real world of applied research, it is probably wrong. More likely, your margin of error is ±8%—double what you thought—giving you a range of 16 percentage points. Yikes, what’s going on? The problem is that margins of error are theoretical, and based on nearly-impossible assumptions about random sampling. They also do not account for non-sampling sources of error.
To estimate the real margin of error, Andrew Gelman, a professor of statistics and political science at Columbia University and one of his colleagues looked at historical polling data. They looked at the theoretical margins of error and point estimates for thousands of polls based on sample size, and then compared these with the actual outcomes of the elections the polls were designed to predict.
Here is what they did, and here is what they found:
We examined 4,221 late-campaign polls — every public poll we could find — for 608 state-level presidential, Senate and governor’s races between 1998 and 2014. Comparing those polls’ results with actual electoral results, we find the historical margin of error is plus or minus six to seven percentage points. (Yes, that’s an error range of 12 to 14 points, not the typically reported 6 or 7.)
Some market research firms will quote you margins of error down to the level of tenths (4.5%) or hundredths (4.52%). Even in a more perfect world, that suggests a misleading level of precision. This new research from Gelman makes it clear how ridiculous that phony precision is.
