Got Too Many Elephants in Your Focus Group?

By elephants, we mean Republicans.  Or maybe you have too many Democrats.  Maybe it keeps going back and forth, which is the problem that Gallup sometimes has.  In the spirit of learning all we can from election season polling, this week we focus on whom to include (or exclude) in your research, analysis, and market projections.

The issue is showcased right now as political polls attempt to measure voter preference and predict the election outcome.  Is voter preference really as volatile and open to persuasion as the polls sometimes suggest?  Probably not.  A 2004 research article in Public Opinion Quarterly carefully documented that much of the volatility in Gallup’s polls results from how they screen respondents and weight their data.  When Republicans are feeling good (like after the first debate) their likely-to-vote scores go way up and Gallup weights them more heavily.  When Democrats are feeling good (after embarrassing Republican gaffes, for example) suddenly they are weighted more heavily versus Republicans.

The problem is a familiar one that all researchers face.  Suppose we are trying to evaluate the strength of one concept over another, or predict future interest and sales in a new product.  Should we include everyone, of just people who are most likely to buy?  How do we even know whether they are likely to buy?  Maybe we should include everyone and weight their responses based on their likelihood to buy?

There are multiple ways to answer these questions, and many options to consider.  When faced with questions like this, here are a few recommendations to help you stay on the right path:

1. Use multiple measures to screen and classify respondents.  For example, pollsters ask about whether potential respondents are actually registered to vote, whether they have voted in the past and how often, whether they know where their your polling places are, and so on.
2. Use validated measures that relate to outcomes.  Many would argue this is where Gallup’s methods fail, because their likelihood-to-vote measure does not line up with what happens on election day.
3. Explore whether respondents differ based on screening criteria, and if they differ, experiment with different weighting scheme(s) that will account for sample biases.
4. Show several outcomes and scenarios using different weighting schemes so that you (and your managers) understand the range of potential outcomes and the factors that will drive one outcome versus another.

It might seem like a no-brainer that focus groups and survey research should include only participants who fit the specific criteria you care about.  But knowing whom to include, designing effective screening questions, and generalizing the findings with appropriate weighting techniques is rarely easy.

–Joe Hopper, Ph.D.