Conscientious and Agreeable Survey Respondents May Be Skewing Your Data

There are three important characteristics of people who “satisfice” on surveys:

1. They are less conscientious
2. They are less agreeable
3. They have different cognitive skills

The first two refer to dimensions of the Big Five personality traits. Conscientious people are efficient and organized vs. extravagant or careless. Agreeable people are friendly and compassionate vs. critical and judgmental. The third refers to people with different cognitive skills who score lower on psychometric tests of memory, numerical, and verbal ability.

A few weeks ago we wrote about survey satisficing, which involves survey respondents “sort of” answering survey questions, but not really. They may pick the first “close enough” answer in a choice set, but not the optimal answer. They may agree or disagree to all questions presented, even though the questions are different and maybe even opposite. They may answer don’t know, neither, or neutral on answer scales even when they do, in fact, lean one way or the other.

Recent research published in Public Opinion Quarterly has shown that survey satisficing is not random, but strongly correlated with psychological traits and abilities. In the words of the social psychologists who conducted the study:

We find large effects of these personality dimensions on the propensity to satisfice in both face-to-face and self-administration modes and in probability and nonprobability samples. People who score high on Conscientiousness and Agreeableness were less likely to be in the top decile of straightlining and midpoint distributions. The findings for Don’t Know responding were weaker and only significant for Conscientiousness in the nonprobability sample. We also find large effects across all satisficing indicators for a direct measure of cognitive ability, where existing studies have mostly relied on proxy measures of ability such as educational attainment. Sensitivity analysis suggests the personality effects are likely to be causal in nature.

“So what,” you ask? Here’s the surprising implication: You should not indiscriminately cut satisficing respondents from your survey data. Satisficing may weaken your measures because respondents are not paying close attention, but this is normal behavior, and it reflects certain personalities and types of people. If you really want to understand your population, you absolutely need non-conscientious and non-agreeable people in your sample, and people with a full range of cognitive skills.

Our advice is to cut truly inattentive and fraudulent respondents from your data, but be extremely careful about satisficing behavior. Erring either way—cutting or keeping—could mess up your results. If you’re not sure of how to walk that careful line between cutting and keeping, give us a call and we’ll show you how.

—Joe Hopper, Ph.D.