Dilbert’s Boss: Focus Groups Are Not Reliable
The pointy-hair boss has a point here, even though he does not realize it. From a research perspective, focus groups should provide rich, new, and surprising depths of insight, not necessarily “reliable” data. In fact, that’s why we typically suggest doing multiple focus groups, in different locations, with different types of participants. We want each group telling us different things from a variety of perspectives. We want each person shedding new light on our research questions, not repeating what others have just said.
William Trochim, a professor at Cornell University and an expert on methodology and evaluation research, has this to say about reliability:
In its everyday sense, reliability is the “consistency” or “repeatability” of your measures. A measure is considered reliable if it would give us the same result over and over again (assuming that what we are measuring isn’t changing!).
That is emphatically what we do not want in focus groups. If they replicate each other, they are a waste. The minute we start hearing things over and over again (called “data saturation” in qualitative research) we stop and move on to quantitative research where reliability becomes essential.
When you have a new product idea, definitely you should test it via focus groups or other forms of qualitative research. Get all the insight and input you can. Use what you learn to revise and improve on the idea. Then your decision to proceed or not should be based on carefully designed—and truly reliable—quantitative research.
Oh, and one more thing. Here’s where pointy-hair boss is wrong: reliable quantitative research is just as affordable as well-done, insightful focus groups. Both are investments worth making!