What Is Forensic Polling Analysis?

Election season polling always highlights the danger of bad pollsters giving the market research and polling industry a bad name. While there are lots of reasons that polls might predict outcomes incorrectly (polling is far more difficult and risky than many people think) there is no excuse for deliberately bad pollsters making up data to support an agenda or asking biased questions to get preferred answers.

It will now take some time for experts in our field to look at who got things right and who got things wrong in this last election cycle, and, importantly, whether anybody deliberately manipulated their data in suspect ways. In the meantime, it brings to mind a bit of “forensic polling analysis” that was done several years ago to identify a bad pollster who was subsequently sanctioned by the American Association of Public Opinion Research.

First, what is forensic polling analysis? The Institute for Social Research’s Center for Political Studies at the University of Michigan defines election forensics as “an emergent means by which to isolate potential anomalies in and diagnose the accuracy of reported election results, using statistical techniques that can identify patterns and assess their chances of occurring by chance.” These techniques can be used to diagnose polling results, as well, hence the term forensic polling analysis.

Here’s the example from several years back: A research firm was accused of falsifying publicly released data, and the firm refused to disclose details of its methods. So a couple of statistical analysts examined the last digit of the numbers released by the polling firm, and calculated the probability of seeing these digits in their specific combinations and frequencies if the data were legitimate. The calculated probability was miniscule: less than one-tenth of a percent, suggesting that the polling data were, indeed, fraudulent.

There are a number of leading industry organizations that have outlined ethical guidelines and principles of disclosure for survey research, including, AAPOR (The American Association of Public Opinion Research) and the Insights Association.

Versta Research believes these guidelines are important to the health and integrity of research.  We adhere to them and encourage our clients to do so as well.  If you are unsure about your ethical obligations when releasing polling data to the public, give us a call and we can help you make sense of the guidelines.

—Joe Hopper, Ph.D.