Slow Down! Your Survey’s Too Fast.
It is possible now to get thousands of responses to a survey overnight so that you can turn around research results as quickly as your managers and clients want. Would you trust the findings? I hope not.
Suppose you want a general population survey that truly represents all U.S. adults. You will need to ensure your sample matches the full population on important demographic dimensions, such as age, gender, region, income, race, ethnicity, and education. With training, experience, and know-how, it is not exceptionally hard to do. But trust me: it takes continual monitoring and adjustment, and it takes time.
In a national sample of 2000 U.S. adults, for example, ideally you want at least one African American man who: lives in the Midwest, is in his 30’s, has a college degree, and earns between $50,000 and $100,000 per year. Census data tells you there are an estimated 75,125 of these guys. It is important to represent them in your sample.
If you want to run your survey tonight, what are the chances that one of them will come into your survey? Almost zero. The 2000 slots in your survey will quickly fill up with the other 99.97% of other American adults. It’s luck if you get him, and not surprising if you don’t. You could work really hard to target him with ads and outreach, but remember you have to do this overnight, and you will have to target hundreds of other specific demographic groups in exactly the same way.
Even Google Surveys, using fully automated targeting and outreach, sifting through billions of users every day, will take 3 full days to deliver 300 respondents who approximately match the U.S. population on age, gender, and region. They don’t even try to match on income, education, race, or ethnicity. It is too hard. It takes human intervention, and it takes time.
So what would you think of a survey of 2000 adults conducted in one day in April 2020 that claims to be representative, and unweighted, because it was fielded with quotas to match Census data on age, gender, household income, race and region? I saw a survey this week that claimed to do this, and my reaction was “Yeah, right.”
When I dug deeper, I was not surprised to see that the full report had bar charts with totals exceeding 100% and margins of error calculated to absurd levels of unachievable precision (hundredths of a percent). I was also not surprised (though disappointed for the paying client) to see that the survey was done by a company with new college graduates who practice consulting skills while operating magic overnight survey robots.
Is it possible to do a credible online survey of 2000 U.S. adults in one day? The Google machines can’t do it. It would take my team at least a week to do it. Ideally I would ask for two weeks so that we had time to sleep. But is it even possible? Actually, with several weeks advance planning (see, it will still take time) and a sizeable budget, the answer is probably yes.
If you’re that rare client with an enormous research budget who can afford several weeks of advance planning for a survey that has to be done overnight, please call us and we will show you how. Otherwise, please slow down. I am 99% sure your super fast surveys are not delivering the trustworthy data you think.
—Joe Hopper, Ph.D.
