Why I Love Political Polls
Actually, Joe Citizen does not love them, but Joe Researcher does. The horse-race polls for media release offer little value to the first Joe because the only poll that really matters is the election itself. But for Joe Researcher, political polls offer unparalleled, ongoing opportunities to understand, validate, and improve our methods of survey research.
That is the reason I eagerly read the just-released Report of the Inquiry into the 2015 British General Election Opinion Polls. Nearly every poll got the election wrong, and by a long shot. It was bad enough to begin shaking confidence more generally in the value of social and market research and the methods underpinning them. So the British Polling Council and the Market Research Society worked with all the major polling firms (carefully reviewing all their raw data and methods) to reconstruct what went wrong and how to fix it for the future.
Here are seven potential problems considered by the council to explain why the polling failed:
- Not accounting for postal voters, unregistered voters, and overseas voters
- Poor wording and placement of vote intention questions
- Late swings in sentiment, with voters changing their minds between the final polls and the actual vote
- Survey respondents deliberately misreporting their vote intentions
- Inaccurate weighting to account for over-reporting of likelihood-to-vote
- Unrepresentative samples
- Problems with mode of interview (phone vs. online)
The culprit? Unrepresentative samples, with “the statistical adjustment procedures applied to the raw data … not mitigat[ing] this basic problem to any notable degree. The other putative causes can have made, at most, only a small contribution to the total error.”
If you do any type of survey work (even if not political polling) every one of these potential problems has a parallel in your own work. For that reason this report is worth reading. It provides a thoughtful and provocative reminder that while we need to think carefully about question wording, response scales, statistical weighting, survey mode, and all the rest – talking to the right mix of people is most important, and easily the most important challenge we face in survey research today.
