Research has consistently shown that a large majority of Americans (upwards of two thirds) do not believe that a sample of 2,000 people can accurately reflect the views of the nation’s population This is mind boggling. It is easy to prove that statistical sampling works, and there are volumes of evidence in our daily lives that it works. Still, most Americans do not believe it.

In some ways, though, it makes sense that they do not believe it. A sample of 2,000 Americans is just one one-thousandth of 1% (.00001) of the full U.S. population. How can this small number possibly predict how everyone feels? In our view, big questions about whether sampling works—and smaller questions about the right sample size for specific research—will likely never go away because even though mathematical proofs and confirming evidence are there, the math and the numbers behind these questions can be quite mind boggling.

Here’s a nice chart showing the maximum margin of error you will get for sample sizes ranging from 1 to 9,999 based on a 95% confidence interval:

There is not much to be gained by increasing the sample size beyond ten thousand, going further to the right. But wouldn’t it be nice to actually show how little we decrease the margin of error by interviewing ten thousand people, 10 million people, or all 310 million Americans?

Turns out, that is nearly impossible to show. The numbers needed for effective sampling are miniscule compared to the numbers of people they ultimately represent. If your monitor is displaying this chart 5 inches wide, the chart would have to extend more than two miles to the right to reach 310 million. It would take you a brisk 40 minute walk to read the whole chart. To show a sample size of 10 million people (a mere 3% of the U.S. population) we would have to extend the chart more than the length of a football field. We thought it would be cool and educational to show you the whole chart. But the mind boggling size of the numbers gets in the way.

What if we adjust the scale, so that all 310 million fit on the chart? The problem then is that the chart shows nothing. It is a vertical line on the left, a flat line along the bottom and not much else, because all the action where sample size makes a difference happens in a tiny slice of the chart that is invisible to the naked eye:

Faced with the impossibility of a meaningful chart displaying the margin of error for all sample sizes, we built an interactive chart that allows a viewer to click and progressively zoom in to the tiny slice of the chart where sample size really matters. The first chart above does that. Move the pointer along the curve to determine the margin of error for any specific sample size. Click and drag the pointer to highlight just a slice of the graph, and it will zoom to that slice. Keep zooming as close as you want, or double click to revert to the original.

This chart does not, however, go all the way out to 310 million. Even with a zoom-able chart, the numbers are too big (Internet browsers cannot handle 310 million data points on one interactive chart). And more importantly, it just wouldn’t matter. That’s the crux of the sampling story. Adding another ten thousand, one million, or ten million people into your sample does not matter. It does not get you much closer to an accurate estimate because you’re already there with just 2,000.

Perhaps an interactive chart like this can make statistics and sampling a bit more engaging for those who feel cursed by math. At the very least, the chart will hint at the mind boggling nature of the numbers, and help you ponder the implications of an efficient and effective sample size. Go ahead with zooming in and zooming out, noting how fast or slow the margin of error changes and the slope of the curve, and consider these key points when it comes to sampling and sample size:

• If you have the budget for a large sample, don’t even consider going beyond a couple thousand, as you gain so little. Beyond a sample size of 2,000 (which gives you a margin of error of about ±2%) you would have to pull an additional 4,700 into your sample (for a total of 6,700) to gain just one more percentage point in precision. The benefit of doing this will almost never exceed the cost of including that many more people in your sample.
  The exception: If you need to understand segments or subgroups within your sample, choose your sample size based on the precision you need for those smaller groups, not the sample overall.
• With a small sample there is a substantial gain in precision for every random person you add to the sample. The difference between a sample of 1,000 and 1,075 is relatively small, decreasing the maximum margin of error by just a tenth of a percentage point. But the difference between a sample of 50 and 125 is dramatic, decreasing the maximum margin of error by more than five percentage points. Another twenty five, fifty, or one hundred respondents can make an important difference when you’re at the left end of the chart.
• The most important decisions you need to make happen in the relatively tiny area between about 100 and 1,200. For each additional 50 or 100 people you include in your sample, you gain a decent improvement in the precision of your estimates. So the questions to consider are always (1) how much precision do you really need, and (2) what is the cost of including each additional person?

In academic journals you will find studies with sample sizes as small as thirty to fifty people. Some of the healthcare research we do relies on samples sizes of fifty to one hundred. Most other research we do, including that for publication in media outlets, relies on samples ranging from 300 to 1,200. As for what others consider to be appropriate sample sizes, we scanned for a handful of recently published public opinion polls:

• An Associated Press (AP) national poll conducted last month in March used a random sample of 1,229 U.S. adults.
• A Reuters poll a few weeks ago (in March and April) relied on a random sample of 1,486.
• A Harvard-CAPS (Center for American Political Studies) Harris Poll in March relied on a sample of 2,746 registered voters.
• Gallup’s most recent telephone-based survey reached out to sample of 1,002 adults, splitting the sample with 502 receiving one version of the survey and 500 receiving another.

The chart, by the way, is based on simple, open source java script that can be used to display other types of marketing and market research data. It is an especially useful way to visualize data if you have thousands of data points and want to show the big picture while being able to drill down into details. Here are more examples. Many thanks to Ben Wang for his programming, advice, and consultation.

Of course this chart, as fun as it is, will not tell you exactly the sample size to choose within that range from 100 to 1,200 because crucial external factors come into play as well. The cost-benefit ratio of a larger sample size shifts rapidly from one end of this range to the other. The tipping point will depend on the particulars of your research study, including its purpose, your audience, and how difficult it is find appropriate respondents.

But if only there were a magic number for sample size, whether it be thirty, one hundred, three hundred, or one thousand, right? Well, don’t despair. Try this magic number: (312) 348-6089. We have a great deal of experience choosing sample sizes and consulting with research, marketing, and communications teams on the key questions that need to be answered within constraints of time and budget. As always, the magic is not in any number itself, but in the highly skilled way in which it is deployed and then turned from data into stories.