How to Trap Survey Trolls: Ask Them for a Story
Telling stories is a natural human behavior, but also highly complex and difficult to imitate or automate. This means that, happily, trolls can’t tell stories. And asking survey respondents for an extremely brief story (even just a few words) about their experience related to a survey topic can help you differentiate real respondents from fake ones as you go about the crucial task of cleaning out fraud from your data.
If you are conducting a survey on experiences in fast food restaurants with children, for example, add an extra open-end question like this:
Please share with us your most memorable childhood experience of eating in a fast food restaurant.
You will be amazed at what you get back. Lots of answers that indicate fraud, like these:
- Good.
- I love fast food restaurants.
- This product tastes so good.
- None.
- Excellent survey.
- Hamburger and Coke.
These are trolls and robot answers. They do not answer the question. Most of them are not constructed grammatically to answer the question. Answers like these usually escape the attention of quantitative analysts and auto-coding or sentiment analysis tools, which is exactly what trolls are hoping for.
You will also get decent answers like these:
- Finding two Hamburgler plastic toys in my happy meal box.
- My parents took us for lunch after going to a doctor, and it was the only time we could eat there.
- The time I ordered 2 large size french fries and nothing else.
These are real people; they are good respondents, paying attention to your survey, and answering questions even if they are keeping it brief.
And you will get back a lot of answers like these:
- Nothing memorable.
- We never ate at these restaurants.
- I can’t think of any.
These are short, and it might seem like the respondents are being lazy. But it’s OK if they are lazy. All you care about is whether they read the question and said something in response that captures the fact that you asked for something that survey open-ends rarely ask for—a story.
The only downside is the added professional time it takes to evaluate an extra open-end solely for the purposes of quality review—a small price to pay if you care about the quality of your data. And real survey respondents seem to like the question, too. It gives them a chance to express themselves in a different way from the rest of the survey.
We have been amazed at how effectively an open-end question like this works at identifying bad data. The Versta Research secret power of Turning Data Into Stories is now our magic power we use to fight trolls!
—Joe Hopper, Ph.D.
OTHER ARTICLES ON THIS TOPIC:
How to Find and Eliminate Cheaters, Liars, and Trolls in Your Survey
