What Makes a Survey Scientific?
Surveys run the gamut from silly to serious, and from sloppy to scientific. Occasionally we do something silly if a client insists (though we generally advise against it). But we never do sloppy surveys, and we hope you never do them either. Lean as far as possible towards rigor and science within reasonable constraints of time and budget.
But what does that mean? What makes a survey scientific? Scientific surveys are carefully built with an eye towards validity, reliability, replicability, and generalizability. More specifically in terms of the survey design and process, here are five components of conducting a survey that will move you towards the scientific end of a survey continuum:
1. Questionnaire design. Write questions that are direct, unambiguous, simple, and unbiased. Avoid leading questions. Think of each question as “measuring” something in a dispassionate way, rather than as eliciting an answer to a question you have.
2. Sampling. Define the “universe,” which is the full target population you want to understand. Then consider ways to get a representative group of them into your survey. True random sampling is rarely feasible, but there are good alternatives to consider so long as you think carefully about representation.
3. Data collection. Lay out a systematic and careful process of finding respondents and asking them to fill out your survey. This is essential to avoiding bias and fraud. And unless you have spent many days mapping out your strategy, fast data collection will likely give you an unscientific mess that invalidates your findings.
4. Analysis. Make theory-driven and hypothesis-driven choices about how to calculate even the most basic statistics, including percentages and means. Decide ahead of time which groups or subgroups you should percentage against. Consider whether to weight your data to adjust for any sampling or data collection biases.
5. Reporting. Write neutral statements that convey the story of behind the data, and provide multiple data points to exemplify and prove that your statements are true. You also need to show, and reconcile, conflicting data points. Don’t cherry-pick, and remember that “neutral” statements need not be boring—they just need to be true.
There are some who argue that the social sciences (political science, economics, sociology, psychology, and all the rest) — and therefore surveys — are not “sciences” at all. Maybe they are right. But for sure there is a difference between good surveys, which aim towards the ideals of rigor and science, and bad surveys. Please join us in striving for the good!
