Ten years ago most survey research was done by telephone. Now the majority of surveys are done over the Internet, and a many of these surveys use respondents sourced through online panels.

What is an online panel? It is a group of people who have expressed a willingness to participate in surveys over the Internet. Instead of calling up random people on the telephone and asking if they would do a survey, we reach out to a random sample of pre-screened panelists by e-mail to ask if they would do a survey.

Besides clients and companies who maintain their own lists of customers and prospects who might be invited to do surveys, there are specialized online panel companies who recruit and maintain relationships with millions of survey volunteers. They are able to create samples that closely match the overall demographics of the U.S. as a whole, or of a specific population.

For many years, important questions have been raised about the quality of online panels. Are they any good? Can we get reliable and representative data from them? Do they work as well as phone surveys?

The Advertising Research Foundation has just released findings from a million-dollar intensive study to answer these questions. They fielded a two-wave survey across 17 online panels. We summarize three key findings here.

FINDING 1: Online panelists represent a broad spectrum of the population.

In the past there have been concerns about panelists being a small group of professional survey takers who join as many panels as they can to take as many surveys as possible. You want to be sure that the people who answer your survey questions represent a broad population, not a small group of people who have nothing better to do.

The ARF research shows that panelists represent a broad cross section of the population, including all demographic groups and regions of the country. According to the study:

• The vast majority (84%) of survey panelists belong to a single panel
• Nearly all panelists are diligent, thoughtful, and careful about representing their opinions and behaviors, including those who belong to more than one panel
• There are five and a half million unique respondents available via panels for surveys in the U.S. This is comparable to the estimated number of people who are willing to complete phone surveys

Of course if sample is sourced through more than one panel, it is possible (but highly unlikely) that a specific person could take a survey twice. Versta Research has validating procedures to ensure that we identify these cases and delete the duplicate response.

FINDING 2: The vast majority of online panelists provide valid and reliable data.

The ARF study shows that online surveys produce high quality data, and that panelists give thoughtful, realistic, and true responses. In fact, the study showed that the single most important factor that hurts survey quality is entirely in the researcher’s control: survey length. This is not surprising, as research has shown for many years that data quality from both phone and online surveys declines considerably if surveys are too long.

The study also found that certain types of panelists give higher quality data than others including older panelists (age 50 to 65), those who have positive attitudes towards surveys, those who take more surveys per month, and those who are on more than one panel.

It should not be surprising that having panelists who are engaged and who like taking surveys gives us better data, and it is valuable to have research that affirms this.

FINDING 3: All online panels are not the same.

Online panels often specialize in recruiting and supplying specific types of respondents, such as IT professionals, physicians, or high-net-worth consumers. But even among comparable panels who are supplying the same types of respondents, the ARF study showed that panels can be slightly different on attitude and opinion dimensions, including intentions to act, reactions to concepts, and so on.

Carefully weighting the data to correct for imbalances in age, gender, education, income and/or region helps, but does not eliminate differences. This finding highlights the need to work with panel companies who adopt aggressive best practices when it comes to panel management, and who work hard to recruit cross sections of the population. In the words of the ARF report, “The findings suggest strongly that panels are not interchangeable. Guidelines and transparency about sourcing is needed, [especially] when blending samples or [when] multiple panels are used to fulfill sample requirements.”

The Bottom Line.

The analysis of the ARF data is still underway, and no doubt there will be additional findings and much discussion about its implications. No research method is perfect, and nearly always you need a smart and experienced team making decisions about which online panel to use, how to manage the process, and how to work with the data once you get it. But so far the research provides good news for those who rely on Internet surveys for insight.

Do you want to be a survey panelist?

One of Versta Research’s partners for online survey respondents is Survey Sampling International, a company that has been in the business of survey research from before the Internet was around. You may volunteer to be a member of their panel at: www.surveyspot.com