New York Times Shifts to Online Polling
Over the last few years it has become clear that random digit dial telephone surveys—once the gold standard of survey research—are no longer so golden. But the New York Times has been a holdout among media giants, refusing to use online polls or to report findings from them. They have clung to the false idea that phone surveys still rely on probability samples.
No longer. The New York Times is now partnering with an online panel company to track and report on political races for the upcoming 2014 election season. Unfortunately they have not done a good job outlining their rationale for the shift, except to say that “a deluge of cheap partisan polls has swamped a shrinking number of high-quality, nonpartisan surveys, making it hard to know who is really ahead in many political campaigns.”
So, we’ll fill in the blanks here with a quick outline of the rationale for dumping phone surveys:
1. Phone response rates have plummeted. In just fifteen years, response rates for rigorously executed phone surveys have dropped from 36% to 9%. This means that from a time and cost perspective phone surveys are no longer feasible. It also means that phone samples are more like opt-in convenience samples than ever before.
2. Probability sampling is a myth. Plummeting response rates mess up the critical premise that phone surveys rely on probability samples. A probability sample requires that we select whom to include in the survey so that every respondent has an exact, known probability of inclusion. But most people won’t answer their phones when we call, and if they do, they won’t talk to us. The probability of inclusion ends up being a guess.
3. Online polling works. During the 2012 presidential election season online polls easily out-performed phone surveys by almost two to one. Google Consumer Surveys’ came within 1.6 percentage points of the actual outcome, versus 7.2 for Gallup. Academic researchers have consistently shown that careful statistical work can overcome biases in non-probability samples.
Kudos to the New York Times. Although they have yet to fully embrace the concept and methods of online polling, at least they are quietly moving in the right direction.
