The Best Way to Stop Survey Cheaters
Using surveys to “quiz” people about their knowledge (versus asking them about their attitudes or behaviors) is tricky. First, online surveys make it easy for people to look up answers they may not otherwise know. Second, people want to get quiz answers right, and so they are motivated to cheat.
This is the reason I usually advocate against quizzing respondents on factual questions. But there is new research just published that gives me new confidence that problems with online cheating can be overcome.
Researchers at the University of Houston and Stony Brook University designed a series of online experiments to measure the extent of cheating on factual questions and to test the effectiveness of interventions. Here are some of their key findings from two experiments done with U.S. adult research panels—the kinds of panels that most market researchers and public opinion pollsters use:
- About one in seven survey respondents (13% and 14% in the two studies, respectively) admit to cheating on knowledge questions.
- But cheaters do not necessarily score better than non-cheaters on those knowledge questions. In fact, for one of the two studies, cheaters scored significantly lower.
- Asking respondents to affirm a commitment to not cheating will reduce their cheating by about half.
How do you measure cheating? The experimenters asked:
“Many people struggle to remember facts, even when they know them, and so they get help remembering. When you were answering the factual knowledge questions, did you get help from any other source, such as the internet or another person? (Please be honest, this is for statistical purposes only.)”
The results of self-reported cheating were then cleverly validated via other measures.
How do you extract a commitment from respondents not to cheat? Here is the wording they used:
“It is important to us that you do NOT use outside sources like the Internet to search for the correct answer. Will you answer the following questions without help from outside sources?” (Yes, No)
The authors note that simply asking respondents not to cheat, or imposing a time limit on questions, do not work as well. It is asking for a specific positive action (“clicking yes”) that does the trick because it constitutes a specific behavioral commitment.
And of course even if people do cheat, the second finding noted above suggests that our estimates of knowledge based on the data are not necessarily skewed.
We have used similar techniques to measure and deal with cheating for other types of survey questions, and indeed they work. This new research is one more boost to my confidence in the power of surveys (when correctly done!) to get valid and reliable measures of attitudes, behaviors, and knowledge.
OTHER ARTICLES ON THIS TOPIC:
Tricks for Getting Truthful Respondents
