A persistent misperception in our industry is that technology can (or eventually will) largely automate our work. It reflects a fundamental confusion between the tools that we use versus the work that we do. This confusion was evident in a recent New York Times article about the surprising amount of time that data scientists devote…
The next time you analyze the results of your customer satisfaction or brand loyalty tracking study, and you notice an upward or downward shift, ask yourself this: Is it reasonable to think that certain customers—either the happy ones or the unhappy ones—were more willing to give you their opinions than the other group? If so,…
It’s the data. We want it all. And to be fair, this article is less about Qualtrics, and more about what makes for truly good and rigorous research. It just turns out that Qualtrics is the one tool that makes it more difficult and more expensive to do that kind of research than any others…
Telephones used to be an awesome way to find a random sample of Americans. Nearly all Americans had a home telephone, nearly everybody would answer if you called, and most were willing to answer survey questions to help researchers understand public opinion. In fact, telephones have been so central to scientific survey research that the…
Over the last few years it has become clear that random digit dial telephone surveys—once the gold standard of survey research—are no longer so golden. But the New York Times has been a holdout among media giants, refusing to use online polls or to report findings from them. They have clung to the false idea…
One of the most useful techniques for in-depth qualitative interviewing is to ask respondents not only what they think, but what their friends, family, or colleagues think. It opens a whole new level of conversation and insight because all of the sudden respondents feel free to talk about their understanding and assessment of the world…
I used to get more worked up about survey answer scales than I do nowadays. I didn’t like even-numbered scales, including 1 to 10 scales (no midpoint!) or five point scales (too blunt!) or seven or nine point scales (not intuitive!). I liked zero to ten scales—and still do—because they offer good opportunity for respondent…
It’s been four years since the AMA convened a panel of thought leaders in market research to ponder questions about how social media fits into research practices and methods. An excerpt of our discussion was published in Marketing News in 2010. Last month, another top industry group (AAPOR) released its Emerging Technologies Task Force report…
For the longest time I was puzzled by what it means to “over-fit” a model to one’s data. How can a model describe one’s data too perfectly? Aren’t we trying to get as close to a perfect fit as possible? This one-minute video will show you an example of over-fitting. After watching it, the odd…
Before the days of DATA MINING this was called DATA DREDGING. And for good reasons, because it was considered not a good thing. Almost any dataset will have nonsense correlations, and if you’ve turned on your magical statistical testing tool at the 95% confidence level, five out of every one hundred nonsense correlations will likely…