Suppose you could have a neural network machine-learned algorithm that is 91% accurate in predicting who will buy your product and who will not, just with a computer facial scan. Wow—91% sounds amazing! Don’t buy it, and don’t be a sucker for big numbers until you look more closely at the math behind those big…
A really good qualitative researcher knows how hard (and how frustrating) it can be to get respondents to tell us things we want to know. We have our interview guide, with all our questions laid out in a nice logical progression. But often when we ask questions, respondents have less to say than we had…
Dividing a market into unique segments makes sense. But the statistical methods we rely on for segmentation often result in segments that are strongly differentiated in useless and misleading ways. A recent analysis of Facebook data by The New York Times illustrates this perfectly.
If you conduct a well-designed, well-executed survey of 500 people you probably think (and yes, we may have told you) that your margin of error is ±4%. It’s beautiful. It’s all based on math. And if you ever had that “aha” moment in statistics class, you probably agree that the mathematical proof of it is…
Two audience members at our LIMRA talk (see Public Studies: Advice for the PR Team and Advice for the Research Team) asked questions about biased research findings. Doesn’t starting with the research story and writing dream headlines lead you down a path of finding or confirming what your internal client wants to hear? I suppose…
This familiar sentence opens the lead article of Public Opinion Quarterly’s 2017 special issue on survey research: “Telephone surveys are dead, claim some survey researchers, and should be replaced with nonprobability samples from Internet panels.” Hmm, I’ve said something similar to that in the past, I thought to myself. I guess I’m not the only…
There are no magic numbers for sample size. There is no such thing as a statistically significant sample. Unfortunately, those two words—statistically significant—are bandied about with such abandon that they are quickly losing their meaning. Even people who should know better (the data wonks at Google Surveys should know better, right?) are saying ridiculous things…
If there is one thing I hope you remember from your college statistics class, it’s this: Correlation does not imply causation. This is especially important to remember in our world of big data. Any large dataset will have hundreds of thousands of correlations, but most of those correlations will reflect purely random occurrences that mean…
At first I thought this was the dumbest survey question ever. Why ask people what device they are using to take the survey, when you know from the platform meta-data what device they are using? Yes, even lame self-service survey platforms can (and should be) capturing information about device, operating system, IP address, start time,…
It was so weird, right after the election, to see a media frenzy heaping praise and attention on the one poll that got it wrong. The USC/LA Times poll predicted a Trump win, with 47% of voters choosing Trump and 44% choosing Clinton. Right after the election, media headlines trumpeted its success: “How One Pollster…