Exploring Different Types of Survey Attention Checks
Last week we highlighted new research showing that removing poor-quality respondents from data can actually make things worse. The reason? Poor-quality respondents may represent a unique segment with respect to what you are measuring, such that removing them skews your sample.
But that doesn’t mean you should keep junky data. Instead, it means you should carefully identify inattentive cases, consider your sampling, and take a light touch in removing problematic non-fraudulent respondents.
The best way to identify inattentive cases is to build into your survey various types of survey attention checks. Here are the types of attention checks that we at Versta Research commonly use:
- Instructional Manipulation Checks (IMCs) and Instructed Response Items (IRIs) are designed to assess whether respondents are following instructions carefully. These checks typically involve including statements or questions within the survey that explicitly instruct respondents to choose a specific response option, such as “Select ‘Strongly Agree’ for this item.” By including these checks, you can identify respondents who fail to comply with instructions or exhibit careless response patterns.
- Reverse-Coding Items is used to assess respondent attentiveness by including questions with reversed response scales. For instance, a survey might contain a statement such as “I always vote in elections” with a response scale ranging from “Strongly Agree” to “Strongly Disagree.” By including a reverse-coded item like “I never vote in elections” and observing respondents who select the incorrect response, you can identify those who are not paying close attention to the questions.
- Timing and Response Duration. Measuring response time is another effective attention check method. You can track the time taken by respondents to answer each question or the overall duration of the survey completion. This approach allows you to identify respondents who rush through the survey or spend unusually little time on certain items, indicating potential inattentiveness.
- Memory Checks are used assess whether respondents accurately recall information provided earlier in the survey. These checks can involve asking questions about specific details or events mentioned earlier in the survey to determine if respondents paid attention to the content. By comparing responses to the original information, you can identify respondents who may not have been actively engaged.
Of course when it comes to the problem of respondent inattentiveness, we in the market research industry often constitute our own worst enemy! There are so many terrible surveys that badger and bore respondents with long and tedious grids, poor grammar, incomprehensible jargon, and questions that make no sense. In short, if you end up finding a huge number of inattentive respondents in your survey, you may need to think more about how you are designing surveys in the first place.
—Joe Hopper, Ph.D.