Why Cost Matters with Online Panel Surveys
Cost matters when you choose a sample or panel provider for your survey because there are good panels and bad panels. Bad panels provide survey respondents at cheap prices. But they do a lousy job managing and screening their members. Not surprisingly, a good portion of the data you get from bad panels will likely be lousy.
A recent study entitled “Dirty Little Secrets of Online Panel Research” by one of our industry colleagues described and documented lousy panel management practices of some companies. Mystery shoppers joined and participated in online surveys offered by nearly all of the leading panel companies that most of us rely on. Here are some of the “worst practices” they uncovered:
Over-surveying respondents. Some panels invite members to complete survey after survey, non-stop. In contrast, good panels limit how often a member is allowed to qualify for, and take, a survey.
Burdening respondents with lengthy surveys. Some panels commonly field surveys up to 30 minutes, 40 minutes or even 80 minutes in length. Research suggests that twenty minutes is the limit of what most respondents can reliably do.
Sending similar studies to the same respondents. Some panels invite members to answer the same set of standardized questions in multiple surveys for different products, over and over again. It teaches respondents what to look for and how to answer, which is unfortunately not what you want.
Blasting invitations to finish studies quickly. Some panels over-recruit so that they can close a survey in just a few hours or in less than a day. A better approach is to give members time to participate because it improves representativeness and avoids having panelists repeatedly shut out of surveys.
Targeting surveys to the wrong respondents. Some mystery shoppers were invited to surveys that were obviously inappropriate (a survey just for women, for example) given the demographics they had reported to the panel company.
Fielding flawed and poorly designed surveys. The report describes as “horrifying” the quality of many questionnaires, with mystery shoppers forced to lie in order to continue because no available answers applied to them.
It is worth noting that good panels exist too. Moreover, research shows that careful and rigorous research done through online panels can be as accurate as older methods of surveying. But of course good panels cost more than bad panels.
Versta Research does a good deal of work through online research panels, and we make a point to continually vet, test, and evaluate how good they are. It is an ever changing landscape because of cost pressures, shifts in strategy among panel companies, and industry consolidation. Rest assured that if you work with Versta and need research fielded through a panel, we will recommend a provider that charges a bit more than some others that may be available. But you will get findings and a story you can rely on.
—Joe Hopper, Ph.D.
