M.I.T. Engineer Flunks Stats
You would think that the chancellor of a prestigious school like M.I.T.—a university that prides itself on practical applications of mathematics—would know something about how probabilities work with survey sampling. Apparently not. Quoting a New York Times article about M.I.T.’s survey of students about sexual assault on campus:
M.I.T. asked all of its nearly 11,000 graduate and undergraduate students to take the survey, and about 35 percent did so. Dr. Barnhart cautioned that it was not possible to say how different the results would have been if everyone had taken part.
Well, if the survey was done correctly, it is definitely possible to project the findings to the entire population of students with certain degrees of confidence and probabilities. That’s what sampling is all about. You don’t need all 11,000 students to answer a survey. You just need a few hundred. And then you can calculate intervals around the estimates and state probabilities that the estimates are correct.
Maybe I’m too harsh. It is true that with a 35% response rate, a non-representative group of students may have self-selected into the survey. But almost certainly it wouldn’t matter, and here’s why:
1. Academic research continues to demonstrate that declining survey response rates are not affecting outcomes;
2. Weighting data is easy with a simple comparison of respondents vs. the total student body on demographics and other characteristics that correlate with the measures of interest;
3. Even extremely non-representative samples can be statistically adjusted and analyzed to achieve surprisingly accurate results.
The key to a great survey is not about getting the whole world (or your whole student body) to respond. It’s about careful sampling and fieldwork and then doing the thoughtful statistical work to make solid inferences. For that, we suggest you pass on the M.I.T. engineers and come straight to Versta Research instead.
