Tips for Sampling from Online Panels

Versta Research is a strong advocate for using online panels for surveys.  As telephone usage and technology have changed, phone surveys are increasingly difficult and expensive, and they are not necessarily more rigorous than other methods.

But that doesn’t mean “anything goes” when it comes to fielding market research surveys and public opinion polls through online panels.  Many panels are poorly managed and overused, and some have high proportions of fraudulent respondents.  While conducting good research through online panels is possible, it requires a great deal of effort and oversight from smart people who know what they are doing.

I was reminded of this recently as we worked with a newer panel provider that recruits respondents through not-for-profit organizations. When respondents complete surveys, their sponsoring NFP organizations get donations.  Response rates are high because members are collectively motivated to participate.  But depending on your study, panelists may not represent the population you want to understand.  If your survey is geographically targeted at the local level, for example, chances are high that respondents are clustered into a limited number of social groups, because that is exactly how they were recruited.

It was a reminder, too, that while sampling through social media and social networking can leverage the amazing power of online social networks, it is critical to understand the effect of networks and clusters on sampling.  And it is critical to incorporate that understanding into your statistical analyses.

Before you commit to any type of online study that relies on sample from a panel, we recommend ongoing due diligence about how the panels are constructed and how respondents are deployed.  At the very least:

1.  Find out how respondents are recruited onto the panel.  As in the example above, different recruitment methods may affect your research design and analysis plan, and for some studies you may need to find an alternative.

2.  Find out how panelists are selected for your particular survey.  You need to ensure that survey respondents are broadly representative of the population of interest.  Quick convenience samples or fast polls using routers can mess that up, so be sure to understand the protocols.

3.  Ask for validation data.  Studies show that panel research can replicate the most rigorous methods used by agencies like the Census Bureau and the CDC.  Ask panel providers for evidence that they have benchmarked their techniques for sampling against data provided by these agencies.

For additional questions you might ask (23 more questions, to be exact) we recommend ESOMAR’s 26 Questions to Help Research Buyers of Online Samples.  Or, give us a call at Versta Research and we will  be happy to guide you through the process.

—Joe Hopper, Ph.D.