How to Fix Your Education Bias in Surveys
How many people do you think are college-educated in the United States? If you’re like most clients we work for (whose family members, friends and co-workers nearly all have college degrees) you will substantially over-estimate that number. The answer is a mere 29%. If you include associate’s degrees, then it goes up to 37%. These numbers are based on the most recent U.S. Census data.
Not knowing how few people actually have college degrees could lead you to overlook an important source of bias in your research. Many (maybe most) surveys today over-represent college-educated Americans, and this is true for both online surveys and phone surveys. If you run an online panel study these days, you will typically see 50% to 60% of your survey respondents having a bachelor’s degree or higher. Why? Because Americans without college degrees are harder to reach, and they are less inclined to participate in surveys.
Survey experts have always known this, but never quite realized how important it can be until the 2016 presidential election. National polls came remarkably close to predicting the true popular vote, but state polls in battleground states missed the mark by a lot. One big factor was that state polls were unlikely to account for less-educated voters being underrepresented. And the unique way in which levels of education aligned with candidate support in this particular election made a huge difference.
The poll director at the University of New Hampshire said in the most recent issue of Public Opinion Quarterly:
“We have not weighted by level of education in our election polling in the past and we have consistently been the most accurate poll in NH (it hasn’t made any difference and I prefer to use as few weights as possible), but we think it was a major factor this year. When we include a weight for level of education, our predictions match the final number.”
The solution? Always include a question about education in your general public surveys. Ideally stratify the sampling so you do not end up with half or more of your respondents having college degrees. When you have your data in hand, analyze whether the key outcome measures are correlated with education. If they are, weight the data to bring your sample more into line with the population you are hoping to measure.
So to fix your education bias in surveys, first remember that not everybody (not even most people!) has a college degree. Then: Ask. Stratify. Analyze. Weight.
