What You Lose with Online Concept Tests
Most companies test new product or service ideas with concept tests to evaluate market interest and to refine their ideas. Most of these (like most surveys, nowadays) are conducted online, which offers speed and cost efficiency.
But there is a downside. While people are quite adept at reviewing ideas and answering questions in online formats, new academic research suggests that information presented on computer screens is processed and understood differently from information presented on paper. This could affect the outcome of concept tests, demanding a careful evaluation of methods before and after conducting the research.
The new study, from two computer scientists who focus on human-computer interaction, shows that most people process online information with closer attention to detail, letting context fall away. Conversely, they process printed information with broader attention to context, but struggle to recall details compared to those who read information online.
The implication for market research is that online survey respondents may do an excellent job focusing on a concept’s details. But depending on how complex the material is, they may have trouble making inferences and identifying which among the concepts is best. In one experiment, for example:
“Participants were asked to read tables of data on four made-up cars. They were then asked to pick out the best vehicle based on the stats they saw. About two-thirds of those looking at the information on a paper printout came up with the correct answer, vs. just 43 percent of those looking at the same information on a laptop screen.” (quoted from The Washington Post)
Does this affect how we design concept tests? Absolutely. For one recent study we had a multiple page brochure and a two-page enrollment form for testing. Given the volume of material and the importance of respondents grasping the context of it all, we express mailed printed versions of everything and then completed phone interviews with them instead.
To be sure, most of our concept tests are done online using a standardized set of measures and protocols we’ve perfected over time. But even then, we recommend never using a “standard” method without first considering how each method may affect the outcomes that need to be measured.
