The Dubious ROI of Customer Satisfaction Surveys
A good friend and client likes to joke during every project: “Remember Joe, correlation does not imply causation.” Well, I thought she was joking. Surely everybody with a job in marketing research knows this? Apparently not. Even professors at fancy business schools seem to forget it, as evidenced by this snippet of an article a few months back in one of our most prominent industry news outlets:
A 2017 study of 7,513 customers of a large North American automotive dealership examined the financial benefits of customer satisfaction surveys. Two findings stood out: First, customers who completed a satisfaction survey purchased more than those who did not, even after accounting for the satisfaction rating. On average, sales per visit were $12.18 greater among those who filled out the satisfaction survey. With an average of seven visits per customer, this amounts to roughly $85 per customer. Even if a relatively high average cost of $10 per customer is assumed for the survey, there is a 750% return on the survey investment.
Unless I have no idea what “return on investment” means, the business professor writing this paragraph believes that a correlation between filling out surveys and spending more means that filling out surveys causes higher spending. Most likely, this is a textbook example of the opposite.
When can you infer causation, and when can you not? Philosophers, scientists, and market researchers agree you need three conditions to infer that X causes Y:
- X must correlate with Y. This is the easiest condition to establish, especially with modern statistics. It is so easy, in fact, that we need a constant reminder that correlation does not imply causation.
- X must precede Y. This seems obvious, but it can be surprisingly difficult to establish with data. This is why we often need longitudinal studies to incorporate measures of time into research, and often we need to make assumptions about precedence.
- Other explanations must be ruled out. Perhaps X and Y correlate because they are both caused by Z (a spurious correlation). Or perhaps the correlation is random. Before we can infer causation, we must test and eliminate other plausible explanations for the correlation we observe.
I took a look at the article this professor cited, and am happy to report that the authors do not, in fact, claim that customer satisfaction surveys yield astronomical financial returns. In fact, quite the opposite. They carefully assess all three conditions needed to infer causation and show that customer satisfaction surveys may actually decrease subsequent purchases, especially with repeated solicitations over time.
So if you didn’t learn this in business school, it is worth remembering here: Correlation does not imply causation. And before jumping to absurd and sensational conclusions (for every dollar you spend on CX surveys, you will earn an extra $7.50 in revenue!) think carefully about all three conditions needed to infer causation.
OTHER ARTICLES ON THIS TOPIC:
The “Disney Experience” of Market Research