Obeys the rule

The Answer

Here’s the answer: The first number must be less than the second number, which must be less than the third number. In mathematical notation: a < b < c. How did you do?

Not many people get it right because there are so many rules that can explain the sequence 2, 4, 8. Most people see a pattern, test it once or twice, see confirming evidence, and then state the rule. In 1960, when this experiment was first conducted, only one in five college students who participated in the study correctly identified the rule.

But what differentiated those who got it right? They looked for disconfirming evidence. There are so many possible rules that you are unlikely to guess the correct one until you start eliminating some. To succeed you need to try and try again until you have several sets that violate the rule.

This is the essence of scientific thinking and it is crucial to the research process. We collect data, analyze it, look for patterns, and offer up explanations. But those explanations are likely to be wrong unless we have tested and eliminated alternative explanations.

There are so many possible rules that you are unlikely to guess the correct one until you start eliminating some.

The take-away from this exercise is that great researchers look for disconfirming evidence. Whether you arrived at the right answer in this exercise matters less than your thought process in getting to that answer. As the original authors of the experiment noted: “The point is not that most subjects failed to give the correct rule at their first announcement, but that they adopted a strategy which tended to preclude its attainment.”

So, if you did not test number sets until you had at least one set come up “No,” make this your mantra for the future: Look for disconfirming evidence.

How? When? Where? And what’s this got to do with the daily work of market research? Here are just three suggestions for places to seek disconfirming evidence that will make you a better researcher.

How to Find Disconfirming Evidence

1. Punish test your questionnaires. You should never feel satisfied with one or two successful tests of a survey. No mistakes? Ha. Unlikely. Don’t believe it. For screening questions test every combination of possible answers. Same goes for questions that drive branching logic. With numeric entry boxes, test for valid ranges and for logical dependencies. Test your back buttons; see if data is replaced correctly when you move forward again. Use a Random Data Generator (RDG) to simulate data with a few thousand cases to test patterns of responses you never thought of.

Ideally you need to subject your questionnaire to as much scrutiny and variability as you will get in real fieldwork. Then you should review your data with as much attention to detail as when you are cleaning and coding the final results. If others are testing for you, ask them to provide a full accounting of what they tested and the mistakes that were corrected. If they can’t find disconfirming evidence before declaring that everything is right, ask them to keep testing.

2. Use control variables in your analysis. So you get to the final presentation and show data that first-time attendees to the decennial fundraiser donated less money than repeat attendees. Your hypothesis? Repeat attendees are more committed, so they donate more. But wait, somebody asks about age. First time attendees are probably younger than repeat attendees, and they probably earn less. Maybe that’s why first-time attendees donated less money.

Your hypothesis might be right, but it will be much stronger if you look for disconfirming evidence by controlling for age. Indeed, there are many possible explanations that could explain your findings. Good researchers brainstorm the alternatives and punish test them against the data. If you can find out which explanations are wrong, then your final answer is more likely to be right.

3. Scrutinize your data collection and coding. Whenever empirical findings are super strong or super surprising, good researchers start with the more plausible assumption that they did something wrong. Maybe the sample was skewed, or the panel supplier messed up quotas. Maybe the questions were leading. Maybe the skip logic functioned incorrectly. Maybe the data were labeled and coded in reverse (it happens all the time!) so the findings are really the opposite.

Great researchers seek disconfirming evidence in all those places where humans (and the machines created by humans) make mistakes. So, always review your programming. Confirm that on-screen images match how they are labeled. Review demographics to ensure a match to your population. Cross check tabulations and statistical modeling by running them on a different package (WinCross or R versus SPSS, for example). Punish test the project, search for errors, and look for evidence that the surprise in your data is wrong.

A crucial tenet of scientific reasoning (and therefore of research!) is that no matter how much data and empirical evidence there is, one can never prove a theory to be true. One can, however, prove theories to be false. Therefore, a systematic, painstaking processes of elimination must be as central to our work as theory building, strategizing, modeling, measuring, testing, and storytelling.

Coincidentally as we drafted this newsletter for publication, the editor of the Journal of Marketing Research, a professor in The Wharton School at the University of Pennsylvania, prefaced the October 2015 issue of the journal with “A Field Guide to Publishing in an Era of Doubt.” His advice was this:

“You need to be able to show that you worked just as hard to try to find support for alternative theories as the one you favor.”

This is just as true for those of us sharing and presenting our research findings in corporate settings as it is in the academic world. Great researchers of every stripe seek disconfirming evidence at every stage of the research process. The ingenious little experiment we’ve replicated above is a potent reminder of that.

[1] Many thanks to the New York Times for calling our attention to this experiment and for the idea of simulating this experiment with a web-based interactive tool.