Three Easy Lessons from a Bad New York Times Survey
Surveys don’t get much simpler than this. It is from the New York Times, which I typically admire for its rigorous reporting, thoughtful analysis, and forward-thinking experimentation with data visualization. But even with such simplicity, this survey (published on April 22, 2018) manages to be terrible in at least three big ways:
It asks a terrible question. Out of nowhere, the survey asks, “Is pizza overrated, underrated, or rated just about fine?” Is this supposed to be a metaphysical question about the concept of pizza? I begin to wonder: What kind of pizza — neapolitan, deep dish, white, red, brick oven, frozen? Rated by whom —me, my family, our culture, yelp reviewers, food aficionados? Rated on what — taste, nutrition, fun, cost? Sorry, I cannot figure out what this question means and what it presumes to measure. Regardless, thousands of people happily answered it, as survey respondents always do, no matter how meaningless it might be.
It relies on a terrible sample. The authors report that the data reflect the opinions of “2,903 subscribers who chose to participate.” The verbiage seems to be their way of admitting that all respondents are entirely self-selected, and no effort was made to build a sample representing some larger population. It’s a classic “fun poll” that media outlets have loved for decades, and fun polls are basically useless. But even supposing there were some rigor behind their efforts to sample and recruit, does anyone really care about the opinions of New York Times subscribers regarding pizza?
It uses a terrible chart. This pie chart is a perfect example of what Edward Tufte calls chart junk, which refers to “the interior decoration of graphics … that does not tell the viewer anything new … to enliven the display, [or] to give the designer an opportunity to exercise artistic skills.” Some may think it is cute, but surely “cute” is not the goal of research and analytics. Moreover, beneath the junk, pie charts are among the weakest choice in our data visualization arsenal. Why? Because humans have a hard time differentiating quantities represented as wedges in a pie.
Granted, I’m being too serious. Surely the New York Times knows that this silly survey is nonsense and poorly done. And I realize my bristling at how bad it is betrays my own poor sense of humor. Still, having the NYT research-and-analytics department pump out surveys like this seems to lend this silliness a legitimacy it does not deserve.
The one bright spot is that we can use surveys like this to learn something. They are an opportunity to remind ourselves that surveys are meaningless unless we: (1) Ask respondents clear and unambiguous questions; (2) Carefully select our respondents to represent a population we want to understand; and (3) Present our findings in direct and compelling ways.
OTHER ARTICLES ON THIS TOPIC:
Optimize Your PR—Don’t Do Silly Surveys
Telltale Signs of a Pointless Poll
