Bon Appétit Gorges on Supersize Samples
Every year I enjoy getting an invitation to participate in Bon Appétit’s “most important survey of the year” because it provides so many good lessons for survey research. This year it has me thinking about the absurd e-mail subject line that says: “We need your help to get 100% participation.”
Bon Appétit has 1,442,388 subscribers. Let’s suppose they were trying to estimate the number of readers who want simpler recipes and that the actual percentage is 57%. If they somehow had all 1,422,388 subscribers respond to their survey (uh, good luck!) they would know the true number is 57% exactly with absolutely no margin of sampling error. Awesome.
Suppose they had just 1,400,000 subscribers respond. What would the margin of error be? One one-hundredth of a percent, so chances are the sample-based estimate would be somewhere between 56.99% and 57.01% Still awesome. What if we go lower, and lower, and lower still? Here is what starts happening to the margin of error as reflected in the most likely range of our estimates:
- 1,400,000 56.99% to 57.01%
- 1,000,000 56.95% to 57.01%
- 500,000 56.89% to 57.11%
- 100,000 56.70% to 57.30%
- 50,000 56.57% to 57.43%
- 10,000 56.02% to 57.98%
- 5,000 55.62% to 58.38%
- 2,000 54.89% to 59.19%
- 1,000 53.90% to 60.10%
- 500 52.62% to 61.38%
- 300 51.34% to 62.66%
Look carefully at those numbers. Dropping the sample size from 1 million to ten thousand does not hurt the accuracy of our estimate by more than a single percentage point. If we go all the way down to one thousand, we might be three percentage points off. Will Bon Appétit make different decisions if their survey estimate is 54% versus 57% versus 60%? In most cases the answer is no.
So who cares about getting everyone to respond? Why would anyone try, spending hundreds of thousands (or millions) of dollars getting a perfect count, when a small sample can do just as well? There are exceptionally few circumstances in which researchers ever need a full census of a large population.
What’s up with Bon Appétit, then? That’s another story. As we wrote last year, their “survey” doesn’t really ask for opinions. It seems to be a ploy to collect data for targeted advertising, and for this (understandably) they want millions of us, not just a sample of one thousand. That would be OK if they were honest about it. But calling it a “survey” seems rather unsavory, and it’s the one time of year that Bon Appétit leaves a bad taste in my mouth.
