Video Content Skews Survey Results
I am the Internet user that advertisers dread. I block as much advertising, video, flash content, and java script as I possibly can so that I stay focused on work without all that annoying, flashing, moving stuff trying to grab my attention. This always makes me wonder: Am I the Internet user that market researchers dread? How many researchers use fancy survey technologies that users like me simply bypass?
A recent study published in Public Opinion Quarterly is an important reminder that technology is cool and lets us push the boundaries of innovation with surveys. But it can also lead to systematic biases because there are people who can’t use or won’t use those new technologies.
This study in particular looked at video content embedded in surveys. The data were collected six years ago, and admittedly much has changed since then: people are more facile with video, technologies are more fluid, connections speeds are faster, etc. But other things have not changed: some people block or skip video content, workplace devices may restrict video, and so on.
The researchers used a high quality probability-based panel and found that roughly four in ten survey respondents were unable to view the videos. Another six percent who could view the videos decided to abandon them before answering the questions. More importantly, the inability or refusal to view videos was not randomly distributed. Non-viewers were more likely to be women, younger, less educated, and employed outside the home. The authors concluded that “the inclusion of videos…has the potential to undermine the accuracy of study findings and distort the representative nature of the study sample.”
What’s a researcher to do? Here is what we suggest if you are using video or other technologies that go beyond basic display functionality:
- Punish test your survey on multiple devices, in multiple browsers, and in legacy versions of browsers as well. Consider cutting the technology if it doesn’t work on nearly all platforms.
- Ask respondents whether the technology worked for them. They will tell you the truth, and the data will help you understand any biases or limitations in your findings.
- Monitor data during fieldwork and adjust quotas to account for biases you are seeing. Yes, biases can undermine survey findings, but smart research can often correct for it.
Of course, all of this requires an exceptional level of attention to detail, which, in our view, every one of your research projects should be getting anyway. Not getting it yet? Give us a call. The simple kind—the old fashioned ring on the phone that still works like a charm.
