A vivid illustration of a statistically spurious relationship is this: People who wear hats tend to drive slower. But there is no causal relationship, as hats don’t cause people to drive slower. It’s just that older people wear hats more than younger people, and older people tend to drive slower. That’s actually kind of interesting,…
I recently came across this strange big book, the Reference Manual on Scientific Evidence, compiled by the Federal Judicial Center and the National Research Council. It’s not fun reading, but it provides an enormously useful and fascinating overview of what makes social and marketing research (and other kinds of scientific research, too) sufficiently rigorous to…
A persistent misperception in our industry is that technology can (or eventually will) largely automate our work. It reflects a fundamental confusion between the tools that we use versus the work that we do. This confusion was evident in a recent New York Times article about the surprising amount of time that data scientists devote…
I’ve always felt suspicious of consultants who claim to spend the bulk of their time solving business problems and offering insights and, by the way, they also do market research. Then I get a look at their research, and my suspicions are confirmed. So often the data is a MESS—an absolute mess that baffles me…
The next time you analyze the results of your customer satisfaction or brand loyalty tracking study, and you notice an upward or downward shift, ask yourself this: Is it reasonable to think that certain customers—either the happy ones or the unhappy ones—were more willing to give you their opinions than the other group? If so,…
It’s the data. We want it all. And to be fair, this article is less about Qualtrics, and more about what makes for truly good and rigorous research. It just turns out that Qualtrics is the one tool that makes it more difficult and more expensive to do that kind of research than any others…
I used to get more worked up about survey answer scales than I do nowadays. I didn’t like even-numbered scales, including 1 to 10 scales (no midpoint!) or five point scales (too blunt!) or seven or nine point scales (not intuitive!). I liked zero to ten scales—and still do—because they offer good opportunity for respondent…
It’s been four years since the AMA convened a panel of thought leaders in market research to ponder questions about how social media fits into research practices and methods. An excerpt of our discussion was published in Marketing News in 2010. Last month, another top industry group (AAPOR) released its Emerging Technologies Task Force report…
For the longest time I was puzzled by what it means to “over-fit” a model to one’s data. How can a model describe one’s data too perfectly? Aren’t we trying to get as close to a perfect fit as possible? This one-minute video will show you an example of over-fitting. After watching it, the odd…
Before the days of DATA MINING this was called DATA DREDGING. And for good reasons, because it was considered not a good thing. Almost any dataset will have nonsense correlations, and if you’ve turned on your magical statistical testing tool at the 95% confidence level, five out of every one hundred nonsense correlations will likely…