Why You Should Avoid Numeric Response Scales in Surveys — They Seem Scientific, but They Are Actually Ambiguous and Difficult to Report
If you read our article a few weeks ago about COVID-19, there were probably several statistics we cited that you (or maybe a colleague) misinterpreted. That’s because instead of reporting the percentage of respondents who did or said something, we reported the mean of all survey respondents’ answers to a numeric scale.
We had no choice. The designers of the survey used a numeric response scale, the results of which are devilishly difficult to communicate in any forum except a scientific journal article.
Here are the particulars. We wrote that Americans’ perceived likelihood of losing a job in the next three months was 10%, which was then specified as referring to the mean likelihood reported by all survey respondents. Unfortunately, most readers will read this and think it means 10% of Americans think they will lose their jobs in the next three months. That is not what it means. Here is the question asked in the survey:
What is the percent chance that you will lose your job because of the coronavirus within in the next three months? Please enter a number between 0% and 100%.
The average response was 10%. But how do we translate that into something easier to grasp, like the number of Americans who think they will lose their jobs? It’s hard to say from the survey question. Just one percent estimated a 90% to 100% likelihood … and eek … even as I write, confusion creeps in. The percentages are now referring to numbers of people and their perceived likelihood of losing a job, all mixed together in one sentence.
Here is the full distribution of how they responded:
If you want to report numbers of people, you are stuck not knowing where your cut-off should be, and you are stuck making an inference about how respondents interpret and use numeric scales in different ways.
Here is a better way to find out how many Americans think they will lose their job. Ask:
Do you think you will lose your job because of the coronavirus within the next three months? Please answer yes or no.
You could offer some type of don’t-know option as well. Better yet, stop asking people to speculate on whether they will lose their job, because that is information most of them do not have. Instead ask them how worried they are. My preferred question, if I were writing this survey, would have been:
Then my reporting task is straightforward. I can easily report how many are worried by summing the percentage who select “somewhat worried” with the percentage who select “very worried.”
—Joe Hopper, Ph.D.