60 Million Surveys Is Too Many
How big of a sample size do you really need? A recent article in the New York Times cited the following statistics:
- A small Voice of the Customer (VoC) research company called Mindshare Technologies collects satisfaction data from 175,000 respondents every day. That’s 60 million in a year.
- ForeSee, a small customer experience analytics firm fielded 15 million surveys in 2011.
These numbers are believable. I get a pop-up survey from ForeSee at least two or three times a week.
And it is absurd. Granted, these companies (and hundreds of other similar firms) are collecting surveys for multiple clients. But almost certainly, nobody needs to collect that much survey data from that many survey respondents. Why not?
1. Huge sample sizes do not improve data precision. While there are significant gains in margin of error by increasing sample sizes at the low end of the spectrum, once you get beyond a few thousand, you gain almost nothing.
2. The cost of analyzing so much data is huge. No matter what technology can do, it can’t make crucial decisions about how to assess whether data are valid and how to fix biases in data, nor can it report on them in meaningful ways that go beyond the simplest of queries. Likewise, big computers can mine lots of data, but all that mining requires smart (and expensive) people programming, analyzing, and reporting on it. It’s the reason that firms have so much data with no idea what it might be telling them.
3. It poisons the well for future surveys. People want to share their opinions and are generous in giving their time for meaningful surveys, but every annoying pop-up survey and each invitation on every cash register receipt teaches another potentially generous respondent to say “No, so please stop asking me.” And sure enough, response rates for surveys have plummeted in the last 10 to 15 years.
There is a smart way to field surveys and to collect data and analyze it. Take only what you need. Think hard and be careful about getting exactly what you need. Make continual adjustments along the way. Be happy (delighted, in fact) with your sample size of 800 because chances are it means somebody thoughtfully designed and fielded your survey, and gave you far deeper insight than a blast of 60 million surveys will ever deliver.
—Joe Hopper, Ph.D.
