The Pitfalls of Auto-Coding Text Responses
An issue we continually struggle with at Versta Research is how to automate the research process and leverage new technologies without losing the essence of what good research does. Good research does not report data, build charts, or generate dashboards. It learns, answers new questions, interprets data, and helps users focus on information and findings that are relevant to their needs.
The last couple of weeks we have been working with a group that specializes in coding and tabulating text responses to open-ended questions on surveys. They have tools and technology that undoubtedly make the process easier and more efficient (we have used those tools, and they are impressive). They are also have a singular focus and expertise that is supposed to help streamline the process, cut costs, and improve speed and efficiency.
The results have been mediocre at best, even with human coders working the technology and making the critical decisions. They efficiently and accurately coded each response into one or more buckets. But they created buckets that give hardly any insight into what the client wants to know. “What do you love most about this product?” The coders accurately identified all consumers who mentioned the physical size of the product. But they lost critical distinctions not only about big versus small, but also about size being a constraint (not enough room for a larger product) versus a preference for how consumers wanted the product to look (an aesthetic choice). They got the topic right, but did not answer the question in a meaningful way. So what good was all that coding?
I asked a colleague (highly paid, less efficient) to fix and re-code the data, and I asked her to think not in terms of topics but in terms of answers to questions. She did, and remarked, “You have an advantage, because you know more about this product and what is relevant to the research than the people (and machines) who did the coding.”
Yes, that’s the point, and that’s the struggle for technology and automation. Smart researchers who know the right questions and continually think about how best to answer them with data will always have an advantage. Even when pitted against the best and most efficient technologies, we will always win the insight contest.
There are lots of places where technology is helping us do our work faster, smarter, and at a lower cost. But no matter what the innovators in tools and technology tell you, it makes a huge difference to have smart people with expertise slogging through the data, deciding how to analyze and present it, and transforming that into a story you can use. We, at Versta Research, consistently and substantially outperform machines and outsourced labor, which means that you, the client, win as well.
—Joe Hopper, Ph.D.
