The Age of Algorithms
Doing “statistics” strikes fear in the hearts of many, so how about if we talk about “algorithms” instead? It’s a safer word because most people in the worlds of business and market research never have to take (or fail) a course in algorithms.
Algorithms are central to the work that we do in business and market research, and they are top of mind for us at Versta Research because we have been involved in several data-intensive projects that involve either (a) developing new algorithms for clients, or (b) tools that apply sophisticated algorithms to data in new and exciting ways.
What is an algorithm? It is “a mechanical or recursive computational procedure” (American Heritage Dictionary). New technologies, data capacities, data collection techniques, and speed have indeed made this the age of algorithms. Here are some interesting examples, including a few we’ve seen in the news lately, and a few that relate to our work:
- The best example (of course) is Google. Good search engines are amazingly fast, efficient, and smart algorithms that find relevant materials based on just a few words that you supply.
- Yahoo is using search algorithms not just to find materials, but to make decisions about what to cover in their own news reporting. It used to be that editors made decisions about what is newsworthy; now algorithms do.
- IBM is perfecting a machine that can play (and win) the game of Jeopardy against humans. It involves algorithms that search for multiple meanings of words and facts, and then matches them against an enormous database of information. It assigns probabilities to each possible answer (or question, as the case may be) and responds accordingly.
- Versta Research completed a project developing a matching algorithm (matching people with products) based on survey data that had thousands of data points on each person and each potential product. The process involved mining the data for ways of matching that optimized customer satisfaction.
- We also just completed a project that used convergent cluster and ensemble analysis (CCEA) to explore patient segments using patient chart data. The technique relies on multiple clustering algorithms, generating hundreds of potential solutions and then selecting the optimal one based on a reproducibility algorithm.
What is interesting is that the “basics” of inferential and descriptive statistics are becoming a smaller and smaller piece of data analysis. More than ever, we combine them with procedures that involve data mining, predictive analytics, Monte Carlo simulations, and so on.
What are the implications for research firms and the clients who employ us? The value of having extremely well-trained researchers who are fluent not only in statistics, but in mathematics, modeling, logic, and yes, even algorithms, is essential. And if, on top of that, you have smart people who know how to use, interpret, and tell a story with that data, the value of your market research will really shine.
—Joe Hopper, Ph.D.
