Market research almost always relies (or should rely) on U.S. Census data. The data are rigorous, reliable, and fundamentally important to every business, not-for-profit, and government body in the United States. So if you get a package that looks like this, as a neighbor did last week, here is what you should do: Open it…
If you ever have trouble separating all the hype of new methods and technology from the reality of whether those methods work, turn to the sober and decidedly non-hyped world of academic research. Some of the smartest people in the world build their careers by pursuing scientific research about research methods. They publish their results…
In Turning Data into Stories we pointed out that numbers have no inherent meaning. There are no essentially big or small or significant numbers, for example—they only become big or small or significant within the context of specific research questions and a range of possible answers. The same is true for p-values by which we…
Here’s a useful way to think about statistical significance. When looking at your data, what’s the probability that it looks like something is there, when in fact nothing is there. Randomness in data (because of sampling) often causes illusions. So testing for significance is all about measuring whether the patterns we see in our data…
One critique sometimes heard about NPS scores is that they can’t be stat tested. That is not true. Though I know of no statistical packages that do it automatically, it is an easy calculation you can do on your own. Here’s the math: n = sample size a = % promoters b = % passives…
You would think that the chancellor of a prestigious school like M.I.T.—a university that prides itself on practical applications of mathematics—would know something about how probabilities work with survey sampling. Apparently not. Quoting a New York Times article about M.I.T.’s survey of students about sexual assault on campus:
Readers of this newsletter and blog know how much we value and respect the New York Times. It is one of the few resources we cite regularly, as it always has relevant, authoritative information that informs our work and that of our clients. So my heart sank when I read this description of a new…
For the longest time I was puzzled by what it means to “over-fit” a model to one’s data. How can a model describe one’s data too perfectly? Aren’t we trying to get as close to a perfect fit as possible? This one-minute video will show you an example of over-fitting. After watching it, the odd…
Before the days of DATA MINING this was called DATA DREDGING. And for good reasons, because it was considered not a good thing. Almost any dataset will have nonsense correlations, and if you’ve turned on your magical statistical testing tool at the 95% confidence level, five out of every one hundred nonsense correlations will likely…
Leading practitioners of survey research now agree that online polling and other forms of non-probability sampling can work. The key is to give careful attention to the survey protocols, data collection, cleaning, and back-end weighting so that ultimately the sample being analyzed matches the population you want to know about in as many relevant ways…