What if a survey asks you a question that implies a yes-no answer (“Have you ever …”) but then offers you answer options that are different from what it implied (“often, sometimes, rarely, never”)? New research tells us that it depends on how the survey is administered. If the survey is online or on paper,…
This is the same survey I told you about a few weeks ago. It came back to me again, since I am part of the ABS-recruited probability-based panel that is fielding the survey. It is still one of the worst surveys I have ever seen. This time I took some screen shots: …
Survey questions that have multiple response options with an instruction to “select all that apply” are common in our industry, and for good reasons. They are efficient and easy for respondents to answer. They are efficient to lay out graphically. They shorten the length of a survey in terms of both time and space. The…
The online customer satisfaction survey I received today had me noticing an unfortunate side-effect of the NPS (Net Promoter Score) schema and visual imagery. For those of you unfamiliar with NPS as a way to measure and track customer satisfaction, here is how it goes. In your survey ask: “How likely are to recommend Acme…
Conducting surveys with your own customers is surprisingly difficult anymore. Phone options are dead, and online surveys are tough, because nobody answers or responds anymore. So what’s a researcher, strategist, or marketing professional to do? How about conducting a paper-based mail survey. It sounds crazy, but we just did it, and it succeeded beyond what…
Even the very best survey shops are pressured to do bad work. My recent experience with one bad survey from a fancy survey shop illustrates in multiple ways one of the biggest problems facing market research today: bad surveys via robots, including the newest dreaded incarnation of robots: Artificial Intelligence. The survey came from a…
If you make your survey questions hard, it turns out that people will make a good faith effort to answer them. But they will make mistakes, and you will end up with bad data that lacks reliability and validity. What I find so interesting is that most people will actually try to answer your hard…
Having focused intently over the last several months on best practices for writing surveys (see Five Last Steps in Writing a Questionnaire and How to Write Surveys: 25 Best Practices) I have been paying extra attention to the work of research methodologists who codify best practices and develop protocols for implementing them. One set of…
So why are we asking them to do our jobs? I ask this question a lot when we execute surveys for marketing and business strategists. They’re smart people who want us to synthesize lots of data into compelling stories for their business clients. They love to assemble data into dashboards, visualizations, and grids. But they…
Customizing MaxDiff exercises by piping in text from previous survey answers might seem like a good idea, but analyst beware: if you are estimating individual-level scores using hierarchical Bayes or latent class analysis, you must split the data into subsets before you calculate the scores. Your results will be nonsense otherwise. Here is the scenario…