Tricks for Getting Truthful Respondents
Difficult or lengthy surveys require hefty incentives, which unfortunately makes it tempting for some respondents to lie. We are facing this predicament in a survey right now. The survey requires two people from the same household who share household finances to complete parallel surveys. We are recruiting from an online research panel. Person 1 gets a small incentive for completing her survey, and a big incentive if she recruits Person 2 to complete as well. What’s to keep her from just doing both surveys herself? Not much, except a personal commitment to telling the truth, and perhaps the threat that if we find out, she won’t get her incentive.
But our plan is to intentionally activate a truth-telling commitment via a question early in the survey about the importance of valid responses. The inspiration to do so comes from two recent studies.
The first study was conducted by the federal government, which “found a clever way to make a little extra money last summer:”
Some vendors who provide federal agencies with goods and services as varied as paper clips and translators were given a slightly different version of the form used to report rebates they owe the government.
The only difference: The signature box was at the beginning of the form rather than the end. The result: a rash of honesty. Companies using the new form acknowledged they owed an extra $1.59 million in rebates during the three-month experiment, apparently because promising to be truthful at the outset actually caused them to answer more truthfully.
The second study was conducted by two political scientists and published in Public Opinion Quarterly. It tested survey “warnings” and found that questions like this substantially improved data quality:
We check responses carefully in order to make sure that people have read the instructions for the task and responded carefully. We try to only use data from participants who clearly demonstrate that they have read and understood the survey. Again, there will be some very simple questions in what follows that test whether you are reading the instructions. If you get these wrong, we may not be able to use your data. Do you understand?
Will techniques like these ensure that respondents are not gaming your survey? Not entirely. But the studies suggest you can easily reduce the number who are. Add to that your rigorous data validation protocols (you are validating your data, right?), and you can feel a lot more confident you are getting the truth.
