Beware Bad Sample from Crowdsourcing
Oddly enough, the researchers who most prize scientific rigor (our research colleagues in academia) are the last to have realized that online sampling can be treacherous.
An article published in Wired magazine last week describes academic psychologists as “freaking out” because they just learned that a lot of their data, which they source from Amazon’s Mechanical Turk (MTurk), may be junk. And if it is junk, a surprisingly large number of published psychology studies are junk, too. Last year, the Journal of Consumer Research reported that 43% of the behavioral research it published in 2016 was conducted by crowdsourcing respondents on MTurk.
The concern was prompted by a graduate student commenting in one of his Facebook groups that “the surveys he conducted with MTurk were full of nonsense answers to open-ended questions and respondents with duplicate GPS locations. He said he had to throw out nearly half of the data in his most recent survey, a sharp increase from what he was used to seeing.”
This is not surprising to us, and we hope it is not surprising to most of our colleagues in the marketing research communities. MTurk is a bottom-of-the-barrel choice when you are considering online panels for sample. That is the reason Versta Research has never used MTurk, though we have been intrigued by the possibilities for pre-testing. The cost for MTurk sample is shockingly low, with psychologists reporting they often pay less than one dollar per person. We typically pay five to ten dollars per person from our sources. And even with higher quality sources for online sample, we often find ourselves throwing out one-quarter to one-third of the so-called data.
Years ago I was initially impressed with the enterprising spirit of psychologists, turning to such a risky and innovative new source to conduct social scientific research. Not so much any more, in particular because they seem to have failed some basic tenets of scientific thinking (see A Quick Puzzle for Market Research Brains).
So here is our recommendation: If you are going to use MTurk, use it cautiously and sparingly. Use it for bare-bones testing, and nothing more. And then whether you are using MTurk or an alternative source of online sample, aggressively review and clean your data. A few months back we wrote an in-depth newsletter article detailing ten important ways to do that. You can read it here: How to Find and Eliminate Cheaters, Liars, and Trolls in Your Survey.
P.S. I just read an article in The Atlantic from several months back and learned that some research panel providers (not the ones we use!) rely on MTurk to source the survey respondents they provide back to you. Yikes. Buyers beware.
OTHER ARTICLES ON THIS TOPIC:
