I’ve been reading through a report from the AAPOR Task Force on The Future of U.S. General Population Telephone Survey Research, which I thought might be interesting and provocative, given how oddly reluctant the organization has been to embrace online methods. Only a few years ago AAPOR (the American Association of Public Opinion Research) finally…
Bad surveys bug me. Bad surveys touted in The New York Times bug me even more, as I expect only the best from them. Bad surveys on the front page of The New York Times with ridiculous and sensationalist findings bug me so much that they inspire blog posts. A front page article on July…
This familiar sentence opens the lead article of Public Opinion Quarterly’s 2017 special issue on survey research: “Telephone surveys are dead, claim some survey researchers, and should be replaced with nonprobability samples from Internet panels.” Hmm, I’ve said something similar to that in the past, I thought to myself. I guess I’m not the only…
There is a small academic niche in market research that relies heavily on extremely cheap crowdsourcing for data. But before getting up in arms about the reliability and validity of such data, you should know this: They used to rely heavily on students enrolled in their college classes for data. Remember having to work six…
There are no magic numbers for sample size. There is no such thing as a statistically significant sample. Unfortunately, those two words—statistically significant—are bandied about with such abandon that they are quickly losing their meaning. Even people who should know better (the data wonks at Google Surveys should know better, right?) are saying ridiculous things…
It was so weird, right after the election, to see a media frenzy heaping praise and attention on the one poll that got it wrong. The USC/LA Times poll predicted a Trump win, with 47% of voters choosing Trump and 44% choosing Clinton. Right after the election, media headlines trumpeted its success: “How One Pollster…
Whenever you see those “margin of error” statements on political polls and other published surveys (even in market research industry publications, unfortunately!) take them with a grain of salt. They reflect only one type of potential error in surveys: namely, sampling error associated with sample size. Chances are that the “real” margin of error (if…
How many young African American men support Donald Trump? Not many. But if one of them ends up randomly selected into your survey, and if you use statistical weighting to ensure the sample does not skew too old and too white, then you are in trouble. This is what’s happening to the LA Times’ election…
A surprising trend among our academic colleagues in marketing, psychology, and the social and behavioral sciences, is that they are crowdsourcing respondents for research studies from Mechanical Turk (MTurk). I am still overcoming my own resistance to the idea, as it rubs against Versta Research’s obsessive focus on data quality. Online sampling absolutely can and…
The last presidential election was a stunning win for online polls and a drubbing defeat for telephone polls. It provoked a loud and contentious debate among survey methodologists about the accuracy of polls and about how (and whether) to revise our methods. So here we are again, four years later with technology having moved faster…