Credibility Intervals Are the New Margins of Error
What IS a credibility interval, you ask? It is a term making its way into mainstream market research just as Bayesian statistics are making their way into market research. A credibility interval provides a range of values, calculated using Bayesian statistical techniques, within which a statistical estimate is likely to fall. It is analogous to a confidence interval, which is the traditional and commonly used measure of sampling error in survey research and statistical estimation.
As with a confidence interval, a credibility interval can be a legitimate, compelling, and mathematically rigorous way of expressing the certainty of statistical estimates. Unfortunately it is being used in the same sloppy, inappropriate, and misleading ways that confidence interval and “margins of error” are being used.
For example, this year’s polls from Reuters cite credibility intervals for its estimates in exactly the same way that other news organizations cite the margins of error and describe it as “the tool used to account for statistical variation in Internet polls.” In response, AAPOR (the American Association for Public Opinion Research) issued this recent statement:
AAPOR urges caution in the interpretation of a new quantity that is appearing with some nonprobability opt-in, online polling results – the credibility interval. The credibility interval is not the margin of sampling error (MOE) that the public has come to understand as the statistical uncertainty of probability based scientific polls. Instead, credibility intervals are being used when reporting on some nonprobability polls (typically opt-in online polling), or for model-based inferences such as arise in small-area estimation.
As a member of AAPOR and as an early backer of the AAPOR Transparency Initiative, we applaud the effort and agree that this usage of “credibility intervals” is misleading. But unfortunately nearly all public opinion polling that quotes “plus or minus Y percentage points” is currently misleading, and, in our view, should rarely be used.
–Joe Hopper, Ph.D.
