Why Vendors Have to Do Everything Twice
Or maybe we should say: why vendors SHOULD do everything twice BEFORE their work hits your desk and you send it back because you found errors. When it comes to something as complex and exacting as market research or public opinion polling, there are almost certainly mistakes the first time around. If a company does not have processes to validate data and deliverables, unfortunately those mistakes end up with you.
Recent errors in an economics paper that laid the foundation for Europe’s austerity programs provides a dramatic and painful example, as outlined by Paul Krugman of the New York Times:
Other researchers, using seemingly comparable data on debt and growth, couldn’t replicate the Reinhart-Rogoff results. They typically found some correlation between high debt and slow growth — but nothing that looked like a tipping point at 90 percent or, indeed, any particular level of debt. Finally, Ms. Reinhart and Mr. Rogoff allowed researchers at the University of Massachusetts to look at their original spreadsheet — and the mystery of the irreproducible results was solved. First, they omitted some data; second, they used unusual and highly questionable statistical procedures; and finally, yes, they made an Excel coding error.
Coding errors like this are the ones that most insidiously creep into research, which is precisely why vendors should do everything twice. And not just twice, but twice with different tools. Some examples:
- If using Excel to produce calculations or statistics, pull the data into another statistics program (like SPSS) and replicate the calculations.
- Likewise, if using SPSS to produce calculations or statistics, pull the data into another statistics program (like WinCross or R) and replicate the calculations.
- When writing syntax that constructs new variables from existing data, cross tabulate the new variable against the old one(s) to ensure the logic was correct.
- Proofread (and read backwards) all written text in reports and press releases that cite numbers to ensure that numbers were correctly pulled from the data.
Sometimes doing everything twice feels ridiculous. We have computers to eliminate errors and help us work faster, right? Indeed they do, but complex computing tools also compound and hide simple errors that are easy to find. Once we set up the processes to replicate results and check for errors, it takes very little time. Our customers get flawless (and correct!) insights, and they never even know that we did everything twice.
