Big Question for Big Data: Does It Work?
The New York Times Magazine ran a feature article in June implying that big data and quantitative algorithms are the secret of how Obama won the election. The young, technologically-savvy “Obama-wonks” who toiled in “the cave” writing thousands of lines of computer code, took advantage of huge repositories of data from the likes of Facebook and TV-monitoring companies (who monitor usage at a much more granular level than Nielsen does). The “wizards” then targeted a small number of specific voters with aggressive and cheap advertising that the Romney campaign presumably missed.
Well, thank goodness for smart readers who can see beyond the hyperbolic writing and ask: Did any of it really matter? Does big data work? As one reader, a professor of communications at Cleveland State University wrote:
[It is] a fascinating article about how analysts in the Obama campaign used high-tech precision targeting to win the 2012 election. The pesky problem is that we have no idea if this technology actually influenced voters. For all we know, people were swayed less by targeted advertising spots than by the presidential debates, news of Romney’s 47 percent gaffe or well-publicized improvements in the economy. The article promotes the widely held perception that social media and other new technology can unlock the keys to electoral success, just as earlier writers attributed great powers to television advertising and, before that, radio.”
And remember the other data wonks who were modeling all kinds of economic and social factors, and tracking and weighting polling data with fancy algorithms? For months before ad targeting even began, these other data analysts were saying that the data already pointed to an Obama win.
The only way to know whether predictive algorithms work is via experiments. Last week we had exactly this conversation with a client for whom we developed an algorithm to score prospects on the likelihood of buying. To know whether the algorithm really works they need to:
1. Withhold targeted marketing from half of the best prospects, and then compare the buy rates of those who got the targeted marketing versus those who did not.
OR
2. Use targeted marketing on two groups: one selected via the algorithm, the other selected according to whatever method they might have used to select without an algorithm.
If the two groups in the experiment have the same (or even similar) buy rates, then despite how fancy and predictive it may seem, the algorithm is not worth it.
Big data, predictive analytics, and algorithms are all the rage these days, and we’re glad that statistics and data analysis (our bread and butter) are now sexy and the stuff of great stories. But big data can’t do everything. It is always worth asking when and where it actually works because there are times when you are just as well off relying on Dilbert’s “copper manager” as on data wonks, wizards, and cave dwellers.
