High Response Rates Hurt Data Quality
An irony of survey researchers’ obsession with high response rates is that higher response rates often hurt data quality. How can that be? It happens because aggressive recruiting boosts the participation of people who provide less reliable information. Two academic articles published in a special issue of Public Opinion Quarterly on “total survey error” nicely review the literature on this topic and provide additional evidence as well.
As much as some of us love surveys, there are others who are just not interested and not motivated. If we beg and cajole them into participating, what happens? They often pay less attention to the questions being asked. They often give less thoughtful, complete, or truthful answers. If they’re smart, they discern patterns to follow up questions, and answer in ways that avoid them. (If I say “Yes, I know that brand” they will ask me 20 more annoying questions about it, so I’ll say “No.”)
As one of the authors states:
Higher response rates are commonly assumed to be associated with higher survey quality. Nevertheless, increasing response rates without attention to response quality may result in an increase of measurement error if it is associated with reluctance to participate in surveys.
Of course, nobody suggests we throw up our hands and take whomever we can get. Careful sampling is serious business, with non-response biases having real and measurable effects on research outcomes.
But these two articles are well worth reading for a sobering reminder that, very often, the harder we work to boost our response rates, the more likely we are to increase error in measurement. For every project, we need to make thoughtful and specific decisions about whether this is the right trade-off to make.
OTHER ARTICLES ON THIS TOPIC:
High Response Rates May Hurt Your Survey
