AI (Artificial Intelligence) Just Rescued Our Survey — We Overlooked a Huge Mistake and a Robot Flagged It for Us
I write a lot about how robots are messing up market research. But yesterday, I had the happy experience of a robot helping us in a surprising a way. It flagged an error in a list of competitive brands for an ATU study (Attitudes, Trial, and Usage)—an error that would invalidated our results if we had not fixed it in time.
Here’s what happened. Strangely enough, I must give credit to Google Surveys.
Our client sent a list of competitive brands to test along with their own. There were errors in the list, so we fixed them. But in fixing them, we made an error as well. Nobody noticed the error in multiple rounds of revision to the questionnaire, and everyone signed off.
As a gut-check for our sampling plan and expected incidence, we decided to run a single stripped-down version of one question through Google Surveys before running our full survey through an expensive panel. If you have never used Google Surveys before, I can tell you this: It is useful for simple one-question surveys targeted to the full adult population, but not useful for much else. The Google Surveys machine submitted the question “for review,” and a bit later sent me a robot message telling me our survey would not be fielded until we fixed one of the brand names being tested.
I was truly annoyed. The brand name it flagged is an unusual name — so I’m thinking: Of course it can’t find the name in a dictionary! I looked up this brand myself and fixed what our client had sent! I wanted a human, not a robot, to understand this, and let me field my survey question!
I did find a human. He looked at what the robot was saying, he looked up the brand and encouraged me to look it up again as well, and sure enough the robot was right. I felt embarrassed and humbled for having made a new error on something I had purposefully set out to fix.
But more importantly I was amazed to see artificial intelligence improving my work, not just messing it up. This was not a simple spell check against a database. It was a robot “understanding” what I was up to, and pointing out a mistake. I had even included some phony brands in my list to test whether consumers would overreport brand awareness. And I was impressed that the phony brands were not flagged as errors.
Despite all my cautions about artificial intelligence and robots adding new levels of fraud and difficulty to the work that we do, and despite my occasional critiques of Google’s Surveys and the truly ridiculous “insights” they purport to offer — I admit that here they were serving me well.
It demonstrated for me the potential for these new technologies to benefit, not just harm, the work that we do in market research.
—Joe Hopper, Ph.D.
