The most thoroughly tested and validated survey questions ever on the planet are probably the 2020 Census questions on race and ethnicity. Census Bureau researchers have spent tens of millions of dollars over the past decade interviewing hundreds of thousands of Americans about race.

As a result, the Census Bureau has implemented a few changes for the 2020 decennial census, one piece of which involves documenting the race and ethnicity of every person in the United States. Does it mean you should be replicating the Census questions in your own surveys?

Possibly yes, probably no.

To help you sort through the issues, this article describes recent changes to the 2020 Census and helps you assess how the new race and ethnicity questions will likely affect your own research and survey work in the future.

Right now the 2020 Census questions on race and ethnicity may well be the most thoroughly tested and validated survey questions ever on the planet.

Here is what we cover:

1. Race and ethnicity are asked differently from years past. We’ll show you the old and the new, and explain why Census Bureau researchers had hoped for even bigger changes.
   
2. The new questions are complex, just like the old ones, which some people find confusing. But survey respondents are resilient. We’ll provide a brief overview of how the questions were validated.
3. You will have to incorporate the new questions into your work. There is no choice if you want to reference census data. But should it change your approach to asking about race and ethnicity in your own surveys? We’ll show you our recommended approach.

No, the Census Race & Ethnicity Questions Are Not That Confusing

We’ll start with the second item noted above, because it was a complaint by an industry colleague that prompted us to begin looking at the issue more closely for this newsletter.

Here is what this colleague posted on a market research discussion forum:

The race question is very odd. I don’t think any of us in the research community (quant or qual) would have asked the question this way. The question groupings and listed categories are very odd. And the required open-end is bound to get a bunch of garbage. How on earth will they analyze the open-ends and find meaningful, quantifiable data?

Researchers certainly can act like know-it-alls when it comes to designing surveys (even the ones who don’t really design surveys). I can sympathize, because I, too, am prone to critiquing other people’s surveys. Moreover, I agree that the new race and ethnicity questions are not ideal.

But wait. It turns out the researchers at the U.S. Census Bureau agree as well, so let’s give them the credit they deserve. They know tons more about how to ask these questions than any online critics in the market research industry. In fact, they spent over ten years and millions of dollars testing their questions. Right now the 2020 Census questions on race and ethnicity may well be the most thoroughly tested and validated survey questions ever on the planet.

The Census Bureau conducted the largest quantitative effort ever on how people identify their race and ethnicity.

Way back in 2010 the Bureau conducted an experiment to begin preparing for the 2020 Census. Their experiment was “the largest quantitative effort ever on how people identify their race and ethnicity.” It involved experimental questionnaires mailed to a sample of 488,604 households during the 2010 Census. Then they called over 60,000 of those households to re-interview them by telephone. In addition, they conducted 67 focus groups across the United States and in Puerto Rico with nearly 800 people. A detailed report of their methods and findings are available in a 151-page report.

Even before that effort, the Bureau conducted one-on-one cognitive interviewing to test for misunderstandings or misinterpretations of the wording. Here is what that involved:

The protocol for the cognitive interviews combined verbal think-aloud reports with retrospective probes and debriefing. Each cognitive interview involved two interviewers working together, face-to-face with the respondent. One interviewer read the RI questionnaire as if he or she was conducting the actual phone interview. Meanwhile, another interviewer observed the interview, took notes, and later asked cognitive interview and retrospective debriefing questions after the RI questionnaire was complete. The cognitive interview and probing questions aimed to explore respondents’ understanding of the race and Hispanic origin questions, their typical response to these questions, any variation that they might have in reporting race and origin, and their sense of burden of the interview. The debriefing probes were semi-scripted, allowing the interviewer to probe on things that occurred spontaneously while also covering a set of required material.

Then in 2015 the Bureau launched another round of research. They tested question format (one question vs. two), response categories (including Middle Eastern or North African, for example), instruction wording (using phrases like, “select all that apply,” for example), and question terminology (such as race, origin, ethnicity, etc.)  to test alternative versions of the question, and to understand any potential problems with mode effects (mail questionnaire vs. online vs. in-person). They sampled and mailed to over 1 million households, and selected about 100,000 of those households for follow-up phone interviews. You can read about their methods and findings for this phase of research in a 380-page report.

If only we could all test our survey questions this thoroughly and so painstakingly!

It is true that the current ethnicity and race questions are not ideal (see the next section for why). But it is absolutely not true that the results will be “garbage.” At this point, Census Bureau researchers know a great deal about how people respond to the questions laid out on the census forms. And they know how to code and tabulate the responses to get a valid and reliable profile of our population.

The New vs. the Old

Wondering what all the fuss is about? Here are the new 2020 Census questions on ethnicity and race:

Compare this with 2010:

There is one reason neither version is ideal, and it has nothing to do with open-end boxes. It’s because Hispanic or Latino is asked separately from race, so when Hispanic/Latino respondents get to the question about race, many of them will select “other.”

Census researchers know this. They strongly recommended that Hispanic/Latino be integrated into the race question for the 2020 Census. But they ran up against a 1997 law requiring federal agencies to specify five minimum categories for data on race (American Indian or Alaska Native; Asian; Black or African American; Native Hawaiian or Other Pacific Islander; White) and two categories for data on ethnicity (Hispanic or Latino; Not Hispanic or Latino). The recommended changes to this law were ignored by the White House, and unfortunately Congress did not step in to fix it.

There are, however, some other interesting changes that did not require changes to the law. First, the word Negro has been dropped. Second, most respondents are now asked to specify their origins along with their race. The idea is to capture a more nuanced portrait of which countries, tribes, and regions of the world Americans of all races come from.

How to Ask Race on YOUR Survey

For several years we wrote surveys that replicated the Census’ two-question format asking for race and ethnicity, and we recommended that clients do the same. The reason is that usually you (we) must weight your data to Census benchmarks to ensure representative samples. Or, we need to compare your data to those benchmarks to gain a deeper understanding of your markets.

But clients are just like other Americans! Few of them think of “Hispanic or Latino” as being a category separate from race. So in our analysis and reporting we found ourselves nearly always recoding the data into a single race variable, and then applying that same recoding to Census numbers to make our comparisons.

Over the last few years, then, we landed on this version of a race/ethnicity question for our surveys:

Note that the response options are in alphabetical order, except for the last two lines. We have found that this question works extremely well. Few people use the “other” category. It doesn’t offend or confuse people. And then with just a little bit of simple analysis and coding work, it easily maps onto current and future Census data:

This is how we now recommend most clients ask about race in their surveys, with some exceptions. If you need to track against your historical data, it may not work. If your customers are unique, varied immigrant populations (like one of our Chicago clients) you may want different details.

Whatever you do, think carefully about your approach — as carefully and thoughtfully as our colleagues in the Census Bureau have done. Although you cannot test as exhaustively as they have done, you don’t need to. The Census Bureau has provided an amazing foundation of knowledge and data you can feel confident relying on for your own work.