Here’s a useful way to think about statistical significance. When looking at your data, what’s the probability that it looks like something is there, when in fact nothing is there. Randomness in data (because of sampling) often causes illusions. So testing for significance is all about measuring whether the patterns we see in our data…
It used to be that online surveys were dismissed as a form of polling because only a fraction of the U.S. population had access to the Internet. That is no longer the case, and I was surprised by the latest numbers showing just how extensive Internet access has become. Almost nine out of ten (87%)…
You would think that the chancellor of a prestigious school like M.I.T.—a university that prides itself on practical applications of mathematics—would know something about how probabilities work with survey sampling. Apparently not. Quoting a New York Times article about M.I.T.’s survey of students about sexual assault on campus:
I recently came across this strange big book, the Reference Manual on Scientific Evidence, compiled by the Federal Judicial Center and the National Research Council. It’s not fun reading, but it provides an enormously useful and fascinating overview of what makes social and marketing research (and other kinds of scientific research, too) sufficiently rigorous to…
The next time you analyze the results of your customer satisfaction or brand loyalty tracking study, and you notice an upward or downward shift, ask yourself this: Is it reasonable to think that certain customers—either the happy ones or the unhappy ones—were more willing to give you their opinions than the other group? If so,…
Telephones used to be an awesome way to find a random sample of Americans. Nearly all Americans had a home telephone, nearly everybody would answer if you called, and most were willing to answer survey questions to help researchers understand public opinion. In fact, telephones have been so central to scientific survey research that the…
Over the last few years it has become clear that random digit dial telephone surveys—once the gold standard of survey research—are no longer so golden. But the New York Times has been a holdout among media giants, refusing to use online polls or to report findings from them. They have clung to the false idea…
Readers of this newsletter and blog know how much we value and respect the New York Times. It is one of the few resources we cite regularly, as it always has relevant, authoritative information that informs our work and that of our clients. So my heart sank when I read this description of a new…
Leading practitioners of survey research now agree that online polling and other forms of non-probability sampling can work. The key is to give careful attention to the survey protocols, data collection, cleaning, and back-end weighting so that ultimately the sample being analyzed matches the population you want to know about in as many relevant ways…
Every year I enjoy getting an invitation to participate in Bon Appétit’s “most important survey of the year” because it provides so many good lessons for survey research. This year it has me thinking about the absurd e-mail subject line that says: “We need your help to get 100% participation.” Bon Appétit has 1,442,388 subscribers. …