How to Avoid Embarrassing Data Gaffes Like This CMO Survey
This survey was done by a fancy business professor, in an endowed chair, at a top-ten university, along with a managing director at one of the world’s largest consulting firms. Then it was printed in one of our industry’s top publishing outlets.
It takes less that a minute to scan this data and to realize it can’t be right. Read the column headings. Every number in the right-hand column should be greater than or equal to the corresponding number in the left-hand column. Half of them are not. Plus the column of numbers on the right should add to 300% and without even doing the math it obviously falls short.
We have some guesses about what’s wrong and how this table needs to be fixed, but put that aside for a moment. The “findings” here represent an embarrassing failure of quality control at multiple phases of the research process, and serve as a reminder that every research group needs to have a quality-check process in place. At Versta Research, we recommend at least the following:
Cross check your data tabulations. Programmed data output and tabulations must be checked for errors. Every row, every column. Ideally, check them against a different source of statistical output. Assume you have errors; build a systematic process to find them.
Logic check your data analysis. As you read through and think about statistical output, look for patterns that should or should not be there, and do this before you focus on what the data says. In the table above, simply focusing on the logical relationship of the two columns makes the error obvious.
Copy check your final deliverables. Have somebody other than you proof read your copy, verify numbers, and fact check the claims and inferences you make. The article accompanying this table specifically noted that the numbers in the column 2 are “aggregating across all three ranks.” A good fact checker would have questioned whether this statement was correct.
Proofing text and cross-checking numbers can feel tedious and time-consuming, but it is not as hard as you think. It took a one-minute glance at this table for us to realize the numbers were wrong.
All it takes to avoid embarrassing data gaffes like this CMO survey is to implement a simple but rigorous quality check process. In no time at all that process will become a habit of your team, and indeed a habit of your mind.
