Why You Should Avoid Numeric Response Scales in Surveys — They Seem Scientific, but Actually They Are Ambiguous and Difficult to Report

If you read our article a few years back ago about COVID-19, you may have noticed some statistics we cited that seemed odd and open to misinterpretation. That’s because instead of reporting the percentage of respondents who did or said something, we reported the mean of all survey respondents’ answers to a numeric scale.

We had no choice. The designers of the survey used a numeric response scale, the results of which are difficult to communicate in any forum except a scientific journal article.

For example, we wrote that Americans’ perceived likelihood of losing a job in the next three months was 10%, which was then specified as referring to the mean likelihood reported by all survey respondents. Unfortunately, many readers will see this and think it means 10% of Americans think they will lose their jobs in the next three months. That is not what it means. Here is the question asked in the survey:

What is the percent chance that you will lose your job because of the coronavirus within in the next three months? Please enter a number between 0% and 100%.

The average response was 10%. But how does one translate that into something easier to grasp, like the number of Americans who think they will lose their jobs? It’s hard to say from the survey question. Just one percent estimated a 90% to 100% likelihood … and eek … even as I write, confusion creeps in. The percentages are now referring to numbers of people and their perceived likelihood of losing a job, all mixed together in one sentence.

Here is the full distribution of how they responded:

If you want to report numbers of people, you are stuck not knowing where your cut-off should be, and you are stuck making an inference about how respondents interpret and use numeric scales in different ways.

Here is a better way to find out how many Americans think they will lose their job. Ask:

Do you think you will lose your job because of the coronavirus within the next three months? Please answer yes or no.

You could offer some type of don’t-know option as well. Better yet, stop asking people to speculate on whether they will lose their job, because that is information most of them do not have. Instead ask them how worried they are. My preferred question, if I were writing this survey, would have been:

Then my reporting task is straightforward. I can easily report how many are worried by summing the percentage who select “somewhat worried” with the percentage who select “very worried.”

—Joe Hopper, Ph.D.