For all the high level intellectual work you do in research (statistical work, design work, analysis, or writing) there is just as much brute force effort you need to put into fieldwork. That means finding the people to interview, convincing them to participate, asking them questions, and recording their answers. Even if you outsource, keeping…
Some respondents answer survey questions with the same answer over and over again. They agree to everything, or they give identical numeric ratings to all items. It’s called “straight-lining,” and we often consider this a sign of poor data quality. I was intrigued by an approach to mitigating the problem I learned last week, shared…
A simple technique that will measurably improve your data quality is to offer survey response options that go from low to high, or from bad to good. Start with things like terrible, zero, poor, disagree, and so on. Then move towards spectacular, excellent, ten, and agree.
Difficult or lengthy surveys require hefty incentives, which unfortunately makes it tempting for some respondents to lie. We are facing this predicament in a survey right now. The survey requires two people from the same household who share household finances to complete parallel surveys. We are recruiting from an online research panel. Person 1 gets…
If you found that 10% of your online survey respondents were ignoring your questions, generating random data, and there was an easy way to identify who those respondents were, what would you do? I hope you would delete them, go back to the field, and replace them. But for some reason there are those among…
An irony of survey researchers’ obsession with high response rates is that higher response rates often hurt data quality. How can that be? It happens because aggressive recruiting boosts the participation of people who provide less reliable information. Two academic articles published in a special issue of Public Opinion Quarterly on “total survey error” nicely…
Getting good responses to open-ended questions via online surveys is a challenge. So much so, that new technologies have evolved to probe on responses just as live interviewers used to probe people on phone surveys. But can you get really outstanding open-end responses to online surveys without fancy technology? The answer is yes, it just…
I’ve always felt suspicious of consultants who claim to spend the bulk of their time solving business problems and offering insights and, by the way, they also do market research. Then I get a look at their research, and my suspicions are confirmed. So often the data is a MESS—an absolute mess that baffles me…
It’s the data. We want it all. And to be fair, this article is less about Qualtrics, and more about what makes for truly good and rigorous research. It just turns out that Qualtrics is the one tool that makes it more difficult and more expensive to do that kind of research than any others…
Last week we wrote about social media and market research, and suggested that industry experts agree it is not yet shaking the foundations of how we do research. So what about the other gigantic Internet development of the last decade—mobile technology? It should be shaking the foundations of how we do research. Current estimates are…