Advice for PR Surveys: Avoid Numeric Scales

As much as we love numbers, we find ourselves often advising clients against using numeric scales in their surveys.  A numeric scale is any response format that asks people to give a number within a certain range to indicate the strength of their feeling or opinion.  The insanely popular survey question used to calculate Net Promoter Scores is a good example:

“How likely is it that you would recommend Acme Solutions to a friend or colleague?  Please answer on a scale from zero to ten, where zero means not at all likely, five is a neutral score, and ten means extremely likely.”

There are many good reasons to use numeric scales and many types of research for which numeric scales are optimal.  The NPS scale is good because it has eleven points with meaningful endpoints  and a meaningful midpoint.  Research shows that scales like this can be highly reliable and valid, with sufficient variability to allow for sophisticated statistical modeling.

But if your objective is to use survey data for marketing materials, public relations, news releases, or white papers, numeric scales make things difficult.  They are not easy to summarize in words, and if you want to use charts that tell quick, compelling stories, you will end up having to do something like this:

The problem with this graphic is that the numbers inside the pie chart are confusing, and the words highly willing, not willing, and neutral were never actually used or selected by most respondents.  Somebody wrote the questionnaire and used a numeric scale without first considering how they were going to use and present the data.

Here is the question that was used: “When thinking of your financial investments, how willing are you to take risks? Please use a 10-point scale, where 1 means Not At All Willing, and 10 means Very Willing.” Had this organization been working with us, we would have advised using a scale like this:

• Not at all willing
• Not very willing
• Somewhat willing
• Very willing

Depending on their objectives and the story they wanted to tell, we may have advised including a “Neutral” category as well.  A scale based on words rather than numbers would have been much more useful in talking about how investors are willing or not willing to take risks.

There are no “magic scales” or response categories that should always be used.  If you find a research professional arguing otherwise, chances are they are not listening carefully to what you need, nor are they thinking much about how the data they collect will deliver on the core objectives of your research.  Telling a story with data requires thinking about the very last endpoint (presentation of data to the audiences you want to reach) from the very beginning (conceptualizing and designing the research).

—Joe Hopper, Ph.D.