How Good Are Online Survey Panels?
Ten years ago, surveys through online panels were rare. Mostly we conducted telephone surveys. Today it is the other way around. With online survey panels being a $2 billion industry in the U.S., there is now a wealth of comparative data to measure and assess the implications of using online surveys as a substitute for phone surveys.
The American Association of Public Opinion Research (AAPOR) convened a task force in the fall of 2008 to study online survey panels, and they have just released their report. It summarizes issues related to recruitment, panel maintenance, post-survey statistical adjustments, validity, and reliability.
Here is a summary of their conclusions and recommendations (quoted verbatim):
- Researchers should avoid nonprobability online panels when one of the research objectives is to accurately estimate population values.
- There are times when a nonprobability online panel is an appropriate choice, as there may be survey purposes and topics where the generally lower cost and unique properties of Web data collection is an acceptable alternative to traditional methods.
- Users of online panels should understand that there are significant differences in the composition and practices of individual panels that can affect survey results.
- Panel companies can inform the public debate considerably by sharing more about their methods and data.
- Full and complete disclosure of how results were obtained is essential. It is the only means by which the quality of research can be judged and results replicated.
AAPOR’s first conclusion is provocative, and sure to stir up a great deal of controversy. It runs counter to what most research firms are doing these days. In an article next week we will consider this conclusion in more depth, and offer a more nuanced perspective than AAPOR. In the meantime and if you’re interested, you can read the full report. If you wish to discuss it or need help interpreting it, please let us know as we would be happy to help.
—Joe Hopper, Ph.D.
