How to Fix the “Up Means Good” Survey Bias
There is an important directional bias in our culture such that things above tend to be perceived as better than things below. Top is better than bottom. Up is better than down. Over is better than under.
Whatever the reasons for this bias (and there are several compelling theories) recent research shows that it affects how people take surveys. So if your job is to design, draft, review, or even just comment on surveys for your organization, you ought to know how this bias works, and correct for it.
This particular bias is documented and described in a 2013 Public Opinion Quarterly article written by researchers at Westat, the University of Michigan, and the University of Maryland. The researchers conducted six experiments where survey respondents were asked to evaluate organizations, professions, and/or foods in terms of how good or bad they are. The relative and absolute positions of the items on the screen were varied, and other potential confounding factor carefully controlled.
Their results were consistent across all six experiments:
“The position of an item on the screen significantly affected its perceived desirability. Overall, the results indicate that items were rated more favorably … when they appeared near the top of the screen than when they appeared lower down…. The effect of screen position on evaluative ratings may be small, but it is reasonably robust across differences in samples, differences in the items, and differences in the methods for manipulating screen position. The analyses also show that the effect was not limited to any specific age or education group.”
Here are two strategies you should consider to fix it:
- Show only one item that needs to be evaluated per screen so that no item is judged to be better by virtue of where it is on the screen, or
- Randomize the items so that all of them appear in top, middle, and bottom positions just as often as the others.
The “up means good” bias is not necessarily a “bad” thing, and, in fact, one conclusion drawn from the work is that most survey respondents work pretty hard and carefully to provide good answers. Survey respondents process potential ambiguities in survey questions and response scales by filling in the holes with extraneous cues (and biases) to help them make good and valid judgements.
But “up means good” is not a bias you want to encourage (I hope), and it may well skew your results if you fail to correct for it. Fortunately, this is one easy bias to fix.