Does Starting with a Story Bias Your Findings?
Two audience members at our LIMRA talk (see Public Studies: Advice for the PR Team and Advice for the Research Team) asked questions about biased research findings. Doesn’t starting with the research story and writing dream headlines lead you down a path of finding or confirming what your internal client wants to hear?
I suppose it could bias your research, but in reality writing dream headlines is meant to be the business-equivalent of writing out hypotheses in science. You need a clear statement of your research questions and what you expect to find. It is the only way you can design precise methods of inquiry and testing so that you end up with real answers instead of random (and biased) speculation.
And just like in science, you can have all the dream headlines you want. But after that, you need to purposefully design your survey to look for disconfirming evidence. This is the hallmark of scientific inquiry, and it is what makes for truly good research (see: A Quick Puzzle for Market Research Brains).
After you have your dream headlines, then, here is what you need to do: Make it easy for your respondents to tell you the opposite of what you “want” to hear. In other words, make it easy for them to say no.
Here is an example from a recent survey we did. One of the client’s hypotheses (her story—her dream headline) was that since most consumers buy her product only once every ten years, they will exceed their budget in order to get the version they really love.
We could ask: “Yes or no, would you exceed your budget to get the one you really love?” The problem is that people are agreeable and like to say “yes.” So we need to do more. We need to remind them that exceeding one’s budget is a big deal, and we need to make it easy for them to say no.
So first we asked whether they had a budget in mind. If yes, we asked what their budget was. Then we asked “How much would you be willing to spend over your budget in order to get your first choice?” The answer box allowed any number between $0 and $999.
Even then we decided it wasn’t enough. Entering “0” is easy, but not easy enough. We needed as many tools as we could think of to let respondents give us disconfirming evidence. So right below the numeric box we offered a prominent and easy-to-click check box as well:
This gave respondents two easy, prominent ways to tell us they would not exceed their budget.
We started with a story. We worked backwards to topics that would support or refute the story. We designed questions and answer options to ensure data would refute the story if indeed it were false. And we made it easy for respondents to give us data that would refute the story.
We always ask you for your dream headlines, because we want to know your agenda and what’s driving your need for research. But we are deeply committed to turning data into stories that are truthful and unbiased, which is achieved by aggressively looking for that disconfirming evidence.
