Why You Don’t Need Big Data
Some business managers and marketing executives mistakenly believe that “big data” will deliver better insight because of the sheer volume of data now at our disposal. Now we just need the statisticians, the computing power, and the analytics software to sift through it all, right? Not so. The truth is, for most purposes you don’t need a lot of data. You need a small random sample of data.
This is easy to prove mathematically, but less easy to accept intuitively. I was reminded of it this week as Versta analyzes data from a survey that we are replicating from one year ago. Our sample size is 1,500 respondents. That gives us a margin of error of ±2.5%, and sure enough, nearly every number this year is the same as, or just one or two percentage points different from, last year. The numbers are so similar that I pause, and wonder whether they could possibly be right.
The numbers are right, and again, it is easy to verify mathematically. But it’s an astonishing reminder that random sampling works. So what if there are 315,612,088 people in the U.S. generating quintillions of “data points” every day? You can understand all of them with a great deal of precision by focusing on just one or two thousand. Whether you track what people are doing on electronic devices, or ask them a few smart questions about what they’re doing, random sampling works.
Of course big data is not just about volume and sample size—it is also about a qualitative shift in the types of data available to us and in our ability to analyze disparate, low-incidence events. But even so, once you identify the kind of data you need, and figure out how to get it, you don’t need quintillions of it.
How much data do you need? Take a look at Versta Research’s interactive graph for choosing sample size to begin getting some ideas. It provides a fascinating and interactive visualization of how little you gain from huge amounts of data. We invite you to explore it. We also invite you to call us with your questions or if you need any help.