The Dreaded Data Dump Survey Returns
This is the same survey I told you about a few weeks ago. It came back to me again, since I am part of the ABS-recruited probability-based panel that is fielding the survey. It is still one of the worst surveys I have ever seen. This time I took some screen shots:
What makes this survey so bad?
- These are not survey questions. Most of this survey asks respondents to fill out detailed data grids, much like they might fill out a tax return or expense report. Best practices about designing and writing survey questions (things like avoiding biased wording, asking about one idea at a time, and so on) don’t even apply here.
- The requests are too complicated. Take a look at what is requested. They want to know who was with me on my travels, my relationship to them, what ages the children were even if those children are not my own. They want the names of hotels, and how I made reservations at each. How did I pay for each … what was the room rate before taxes … was there a discount applied … what date did I leave on each trip … what date did I get back … the list goes on. Few people will recall such details without pulling up records and receipts.
- It is too long. To fill out just the parts of the survey I’ve shown you in these snap shots, a truthful and conscientious respondent would spend roughly 15 minutes. It doesn’t look like much, but each little boxes requires thought and a response. Moreover, these snapshots show just a small portion of the entire survey. As I navigated my way through, I estimated it would take a reasonable person about 2 hours to complete the entire survey if they were offering honest and thorough answers.
- The respondent gets nothing. Well, not exactly nothing. As a panelist, I am offered rewards points worth one dollar for completing this survey. Seriously, one dollar for two hours of time to do something that is uninteresting and tedious. If I were fielding this study for a client, I would recommend paying respondents $50 to $200, and I would recommend doing it in person to ensure that the data were legitimate. Paying a big incentive, and completing the task in person, is how The Federal Reserve Board fields its Survey of Consumer Finances, which asks for similarly complicated information.
Given all this, I doubt people are really filling out this survey. Maybe robots are filling it out. Maybe people are zooming through to collect their buck. But I am absolutely certain that if the dataset from this fieldwork landed on my desk, I would not trust it, not even a little bit. Skilled and experienced researchers see way too much bad data coming from surveys even easier and better than this.
—Joe Hopper, Ph.D.
