Who’s Your Data Janitor?
I’ve always felt suspicious of consultants who claim to spend the bulk of their time solving business problems and offering insights and, by the way, they also do market research. Then I get a look at their research, and my suspicions are confirmed. So often the data is a MESS—an absolute mess that baffles me for weeks at a time. Is this really how others in our industry do market research?
Any consultant who is really doing research and working with data is spending most of their time in the unglamorous world of data janitorial work. They are cleaning, coding, tagging, writing syntax, formatting, and defining their data. Quoting from a recent New York Times article on this subject:
Handcrafted work — what data scientists call “data wrangling,” “data munging” and “data janitor work” — is still required. Data scientists, according to interviews and expert estimates, spend from 50 percent to 80 percent of their time mired in this more mundane labor of collecting and preparing unruly digital data, before it can be explored for useful nuggets.
“Data wrangling is a huge — and surprisingly so — part of the job,” said Monica Rogati, vice president for data science at Jawbone, whose sensor-filled wristband and software track activity, sleep and food consumption, and suggest dietary and health tips based on the numbers. “It’s something that is not appreciated by data civilians. At times, it feels like everything we do.”
I suspect the dirty secret in our industry is about the number of consulting firms who never do this kind of janitorial work. They have data, but it’s not cleaned up, and it’s mostly garbage. So they leap to unwarranted answers, seemingly unaware that every query of the data requires a thoughtful and sometimes tedious effort to make the data query-able.
So who’s your data janitor? Ask your vendors and find out. The best technology and the best tools will never replace the need for janitorial work. They just make us better, smarter, faster, and more innovative janitors. And then only after investing time and effort on the janitorial side can we turn all that data into sparkling stories that answer your most pressing questions.
