How Polls Pass CNN’s Quality Review

A survey we conducted for a client this year was just vetted by CNN and we were asked to provide all kinds of detailed information about what we did and how we did it so that our client could get news coverage for the survey they commissioned.

This list of 16 questions that CNN will ask before they are willing to publish your polling data is worth reading because somebody at CNN clearly knows their stuff. It is far better than anything I have seen from the New York Times, and it is far better than what the Associated Press (AP) sets as its standard.

It is good because it reflects a keen understanding of the issues we confront in survey research without being methodologically rigid. It addresses issues of survey mode, response weights, representation of a universe, different types of error margins, skews from education biases, and many other crucial issues. Notice also that it never refers to response rates.

It is a primer on all the important issues we need to think about all the time whenever we do any type of survey research. In short, every good researcher should be able to answer every question here, for every project she does.

The CNN Transparency Questionnaire

1. What survey firm conducted the poll?
2. How were respondents interviewed – by live interviewers on the phone, IVR, online, selfadministered questionnaire or another method?
3. Who paid for the survey and why was it done?
4. How many people were interviewed for this survey?
5. In what language(s) were respondents interviewed?
6. Please provide a copy of the full text and interviewer instructions/programming for all questions included in this survey release.
7. When was your survey conducted?
8. What is the source of your sample for this survey, and by what method were respondents selected? Please be as specific as possible, and if via web panel(s), please include a description of how the panelists were recruited. If your study was conducted online and included respondents chosen via routers, approximately what percentage of respondents were directed to the survey via routers?
9. If any quotas were applied to sampling or interviewing, at what stage were they applied, what variables and targets were used, and what is the source of your estimate of the target quota?
10. What is the universe of people you are trying to survey, and what makes you confident that the sample source represents that universe?
11. If surveys were conducted by telephone, what percentage of interviews were conducted via calls to cellphones? If surveys were conducted online, were respondents allowed to complete the survey via mobile browsers, and approximately what share of your respondents did so?
12. If surveys were conducted by telephone, how many callback attempts did a sampled number receive before being retired?
13. If surveys were not conducted by a live interviewer, what do you do to ensure your respondents are real people and are paying attention to the survey?
14. What is your estimate of this survey’s error, how is it calculated, and why is this an appropriate error estimation for your survey? If you are reporting a margin of sampling error, has it been adjusted for design effects?
15. If your survey has been weighted, please list the weighting variables and the source of the weighting parameters. If your survey has not been adjusted for education, please explain why and provide an unweighted frequency for education distribution among your respondents.
16. Is there a minimum unweighted sample size you require before releasing any subset estimates, and if so, what is it?

I am proud to say that Versta Research rarely sees media outlets deciding not to report survey findings from our research for what they perceive as lack of rigor or transparency. This is because we are, indeed, aggressively rigorous and transparent in our methods. We are also painstakingly accurate in how the survey findings are formulated and checked.

Keep all of that in mind as you contemplate your next PR survey campaign. Versta Research can help you succeed by bringing together creativity, insight, experience, know-how, authenticity, rigor, truthfulness, and transparency.

—Joe Hopper, Ph.D.