High Response Rates May Hurt Your Survey

A couple weeks ago we presented new data showing that response rates continue to decline.  You can now expect that a typical, rigorously executed phone survey will yield a response rate in the single digits.

Scientific evidence over the last decade has shown that high response rates do not necessarily yield more accurate surveys.  In fact, it turns out that high response rates can actually hurt the accuracy of surveys.

A recent article in Public Opinion Quarterly adds new evidence to these claims.  In a secondary analysis of data from multiple surveys over the past several years, including random digit dial (RDD) telephone surveys, and online surveys using both probability samples and non-probability samples, the researchers find that higher response rates are correlated with lower accuracy of survey results.  The authors summarize the findings:

Among the seven RDD telephone surveys, response rates were positively correlated with the size of each survey’s average absolute error without post-stratification (r = .47). Completion rates were also positively correlated with the size of average absolute error among the seven probability sample Internet surveys (r = .47) and among the non-probability sample Internet surveys (r = .61). Thus, higher completion rates and response rates were coincident with less accuracy, not more.

What’s going on here?  One theory is that higher response rates are achieved by including respondents who are less interested and less focused on the questions.  It is not that these survey respondents lie; it is that they don’t listen (or read) closely, and thus fail to provide thoughtful and careful answers.

So, yes, there are lots of things you can do to boost response rates.  You can call people over and over, badger and beg, and even offer them money.  You will boost your response rate.  But you may hurt your survey as well.

—Joe Hopper, Ph.D.