One Lousy Question Can Cause Massive Measurement Error
It should be straightforward to ask someone a simple factual question and receive a reliable answer. Questions like: “Do you have children?” “Do you attend church every week?” or “Do you own a bank account?” seem easy and clear-cut.
However, the moment we begin drafting even these seemingly straightforward survey questions, we quickly uncover ambiguity and potential complications. For example, when asking, “Do you have children?” do we also mean adult children? Do we count children who live primarily with an ex-spouse? It becomes tempting to add elaborate specifications to precisely define what we mean.
But here’s the catch: overly detailed specifications often lead directly to substantial measurement errors.
This point was clearly illustrated by two academic researchers in an article entitled “A Simple Question Goes a Long Way,” published in the Journal of Survey Statistics and Methodology. The researchers replicated a series of questions regarding financial asset ownership from a long-standing health and retirement study. Respondents were first asked about owning stocks, then bonds, followed by CDs, government savings bonds, or treasury bills—in that sequence. Finally, participants were randomly assigned one of three versions of a question about bank account ownership:
- Version A (most complicated): “Aside from anything you have already mentioned, do you or your spouse/partner have any checking or savings accounts or money market funds? Please do not include CDs.”
- Version B (moderately complicated): “Do you or your spouse/partner have any checking or savings accounts or money market funds?”
- Version C (least complicated): “Do you or your spouse/partner have any checking or savings accounts? Do you or your spouse/partner have any money market funds?”
The results were striking:
- Version A (most complicated): 74% reported owning a bank account.
- Version B (moderately complicated): 85% reported owning a bank account.
- Version C (least complicated): 92% reported owning a bank account.
These differences are huge! Given Federal Reserve data showing that approximately 94% of adults actually own a bank account, the simplest question (Version C) comes closest to accurately capturing reality. Furthermore, the researchers demonstrated that the causal mechanism is that simpler questions make it easier for people with lower levels of cognitive ability to answer correctly. This is true even when controlling for age, education, and financial literacy.
The practical implication is clear: If you want to minimize measurement error, keep your survey questions super simple. Resist the temptation to overly clarify or insert excessive caveats. If a survey question becomes complicated due to a legitimate need for nuance, break it down into smaller, simpler parts. Ensuring each part is easy, direct, and clear will significantly enhance the validity and reliability of your survey results.
—Joe Hopper, Ph.D.
