Dilbert Does Predictive Analytics

We love this cartoon because it raises such interesting issues about probability and prediction in market research.

Here’s a quiz.  Suppose you own a retail store and want us to help predict which customers that come through the door will buy.  You want to focus your sales efforts on the best prospects.  We do our market research and offer you a fancy “predictive analytics” model.  Alas, it offers zero percent accuracy in identifying which customers will buy.  Would you purchase our fancy model?  Of course you should!

If our model offers zero percent accuracy, then all you have to do is reverse our prediction to get 100% accuracy.  It means that every customer we identify as a buyer is a non-buyer, and vice versa.  So zero percent amounts to the same thing as 100 percent accuracy.  The worst predictive model is one that has 50% accuracy, which is identical to a coin toss.

So here’s another quiz.  Suppose all we do is count the number of customers who come through the door and then count how many purchase.  We find out that one customer purchases for every ten in the store.  So we offer you a fancy predictive analytics model that accurately predicts buying behavior 90% of the time.  Would you purchase our fancy model?  Of course you should not.  We can “build” a model with that kind of accuracy simply by predicting that every customer is a non-buyer.

When you start thinking about probabilities this way, it is less impressive to read claims like “Our predictive analytics model can identify purchase behavior 75% to 85% of the time.”  The question is, “Versus what?”  And, “How often can I accurately predict without the model?”  Seventy five percent might be substantial, and we have no doubt that solid research can help you better predict behaviors and identify your best customers.  But always remember that the copper manager in your pocket can predict a binary outcome 50% of the time, as well.

—Joe Hopper, Ph.D.