The pandemic changed everything: our families, schools, workplaces, health, safety, and prosperity. And it changed the research industry, all the way from what we research to how we research it.
At Versta we always talk about trends we experience and see as we go about our work, and I can tell you the last few years have been remarkable. The pandemic taught us multiple lessons, some of which are sobering. Our feature article, Five Research Lessons from the Pandemic Years, outlines five of those research lessons that we see as having transformed market research during the pandemic years.
Other items of interest in this newsletter include:
- The Problem with “Check All That Apply” Survey Questions
- Mobile Phones Still Present Challenges for Market Research
- Data Visualization Is Not the Silver Bullet You Hoped For
- A Better Way to Ask Race and Ethnicity (and Proof That It Works)
- How to Measure Brand Authenticity
- How to Test Knowledge and Misperceptions on a Survey
- Are They Cheating or Helping? Survey “Cheating” Raises Thorny Issue
- This is What a McKinsey Survey Looks Like
- How to Fix a McKinsey Survey with 10 Best Practices
- Finding Fraud in Public Polls: Our AAPOR Presentation
- Tempted to Try SMS Text Message Surveys? Don’t Bother.We are also delighted to share with you:
… which highlights some of our recent work for clients such as the Alzheimer’s Association, Wells Fargo, the American Academy of Dermatology, Fidelity Investments, and The Standard.
As always, feel free to reach out with an inquiry or with questions you may have. We would be pleased to consult with you on your next research effort.
The Versta Team
Five Research Lessons from the Pandemic Years
There’s no minimizing the toll of COVID-19. It upended lives to such an extent that we now have “pre-pandemic” and soon there will be “post-pandemic” and COVID-19 will be an important defining moment in our history.
In our professional world of research, the shift has been (and will be) dramatic as well. It has affected not only how we work (from what locations, and via what technologies) but the substance of our work as well. This newsletter highlights five of those shifts, which we’ve cast as “lessons learned,” though the implications are still playing themselves out.
1. The Fraud is Frightening
Soon after the pandemic began, research froze. Studies were postponed for a year or longer while clients took a wait-and-see approach in a world that was suddenly upended. The result was that all those people who signed up to participate in research surveys walked away. There were not enough research opportunities to go around. There was much else to worry about and no rewards to be had. The panels who supply hundreds of thousands of people willing to give research input suddenly found themselves empty handed.
Unfortunately that opened up a rich opportunity for fraudsters. Organized, disreputable groups set up workrooms with hundreds of employees to fill panels with real people (not robots) posing as ordinary respondents to harvest the survey incentives. The fake respondents were expertly trained to bypass fraud-detection, which meant that when we started fielding surveys again, the volume of fraud was astonishingly high.
And it had a lasting impact. The fraud we now see is amazingly sophisticated. The “respondents” know how to answer survey questions consistently, and they write in full sentences with credible-sounding open-ends. Now we have to go through a more labor-intensive process (including watching video recordings of on-screen mouse movements and reading open-ends in clusters to detect patterns in cadence and vocabulary) to find and weed out all the fraud.
At the height of the pandemic and for some types of research, we detected up to 70% of research respondents being fraudulent. Sadly, we also saw research studies – some about the pandemic itself – being circulated with findings that were clearly not true.
2. We Need Government Data
Versta turns to government agencies for data all the time, compiling and analyzing data from the U.S. Census Bureau, the Bureau of Labor Statistics, the National Institutes of Health, and the Federal Reserve. The depth, richness, usefulness, and availability of this data is remarkable. Our own primary research (and our clients’) pales in comparison to the treasure trove of what our government creates for us. Please, clients and colleagues, understand that we all need this data. Everything we do relies on it.
The pandemic offered two reminders of how critical this government data is. First, we were quickly confronted with the inherent time lag of the data. With the breadth and rigors of research from government agencies, it takes roughly two years from fieldwork to data availability for the datasets we use. Suddenly there were no valid benchmarks for measures of physical health and financial health that were declining dramatically within our study populations. We had to make best-guess assumptions, highlighting how crucially we rely on gold-standard benchmarks every day for our work.
The second reminder came as the pandemic became linked with a wider political assault on government agencies. The NIH came under attack in some of the same ways that the 2020 Census had been coming under attack. It was deeply depressing. These institutions, built up over hundreds of years, serve and protect our freedoms, health, and livelihoods. We – and you – need these agencies. So much of what we – and you – do relies on them.
3. Keep Your Trackers
Research firms (and probably you, as well) have a love-hate relationship with tracking surveys. On the “love” side, they provide excellent training in how to conduct rigorous, replicable research. They provide a consistent source of valuable work. They teach about the differences between, and sources of, normal variation and real change. And of course, they provide an invaluable historical record of the important measures central to your organization.
On the “not-love” side: They grow enormously in data complexity with each iteration, especially when edits are made. They can get boring. And along with this, they provide dangerous temptations to overinterpret the slightest shifts because clients desperately want and expect to see change, even when cautioned that change is unlikely. Too often we find ourselves having to call non-significant shifts “directional” even when they are not.
The pandemic, however, demonstrated the value of replicable, longitudinal data for organizations beyond just the U.S. Census Bureau and National Institutes of Health. The pandemic extensively changed multiple everyday behaviors for long periods of time. The clients who were already out there measuring and monitoring were able to see these changes clearly and respond to them quickly. Recent surveys we conducted for one of our clients about mental health issues in the workplace provides one example.
4. Strategists Need Our Help
The thing we love about strategists is that they know exactly what they want. By the time they come to us, they have analyzed their business needs. They have figured out what kinds of data will help them make decisions. They know why they want research, how they are going to use it, and how they want to report it.
The problem is when strategists write questionnaires. They get so intently focused on what they need, that they ignore how to elicit valid, reliable, and sensible data from people who actually answer surveys. For example, we see strategists draft surveys with huge multiple-select grids that look like data forms the IRS might design. They ask survey respondents to work their way through endless rows and columns of options, and to think carefully about multiple dimensions and response options all at once. Often they use jargon that stumps even the most educated.
As the pandemic stretched client-side research staff and budgets, we often partnered directly with business strategists. Many of them handed us questionnaires they wanted to use. It was a lot of work trying to revise and reorganize their questionnaires so that our efforts would not fail. For the most successful projects we convinced the strategists to lay out their business questions, not survey questions. Then we designed how best to get the data that would answer their questions.
5. Thought Leadership Matters
Lastly, we learned about the importance of thought leadership. Versta Research conducts a great deal of primary research to support thought leadership, generating statistics to help clients tell their story in ways that get noticed by the consumers and business leaders those clients serve. Often we think of this research as a luxury – a “nice to have” that is second to the strategic marketing research our clients do.
The pandemic taught us otherwise. Even though research budgets froze, we noticed that many of our clients began investing more in thought leadership than ever before. Some ventured into new topics and bigger studies, and they began planning more strategically for ways that research – broadly designed to explore issues core to their mission – could support multiple pillars of their efforts.
It was eye-opening, and it makes sense. Building an ongoing foundation of expertise, and becoming a known and trusted source for that expertise can carry organizations through ups and downs. Versta Research is fortunate that so many of our clients are forward-thinking. During a long and stressful time of illness, loss, shut downs and uncertainty, it kept us busier than ever before.
Stories from the Versta Blog
Here are several recent posts from the Versta Research Blog. Click on any headline to read more.
We sometimes advise against multiple-select questions, believing that respondents give less attention to each item in the list. New eye tracking research suggests we might be wrong.
Cell phone dialing is now essential for phone surveys, just as mobile-optimized surveys are essential for online surveys. A lot of market research continues to ignore this.
It is magical thinking to believe that charts and graphs and fancy new tools for data visualization will solve the challenge of finding insights in data.
Versta Research has a new way of asking about race and ethnicity on surveys, and it is reducing respondent confusion and complaints by 80%.
This article outlines the six measurable dimensions of brand authenticity — established in academic research — that you should include in any survey of brand authenticity.
Surveys are not quizzes, so you need to be careful when writing questions designed to test knowledge. Here are five best practices we recommend.
New research suggests that a third of survey respondents may “cheat” by looking up answers. But that finding likely reflects poor survey design.
This example of a survey from a high-end business consulting firm is proof you are better off hiring a specialized marketing research firm.
Lest we get too carried away with criticism, here are some solution-oriented learnings from the mess-of-a-survey we showed you last week.
Versta Research presented at the 76th annual AAPOR conference with a presentation on new methods to detect fraudulent survey respondents.
Some providers are now trying to sell survey capabilities that work just like mobile text chats. New research shows there are no advantages.
Versta Research in the News
Versta Research fielded a survey of 3,569 working households to support Fidelity’s 2023 Retirement Savings Assessment. The results provide a snapshot into the nation’s retirement readiness based on workplace and individual savings accounts, Social Security benefits, pension benefits, inheritances, home equity, and business ownership. Findings have been featured on Fox Business and Investopedia.
The Alzheimer’s Association commissioned a new survey from Versta Research for its 2023 annual Alzheimer’s Disease Facts & Figures report. The survey documents current patient loads and future shortages among neurologists, geriatricians, emergency medicine specialists, and neuropsychologists in providing dementia care. Findings from the special report have been broadcast on ABC, NBC, CBS, and Fox News outlets nationwide. A video summary is available as well.
Versta Research conducted a survey for Wells Fargo of young adults who are in line to inherit at least a million dollars in the future. Survey findings are highlighted on a special microsite from Wells Fargo Wealth & Investment Management, and available in a full report.
The Standard launched an online resource center for employers called Behavioral Health in the Workplace that features two years-worth of new research based on work by Versta Research. It includes a full array of blog posts, e-books, whitepapers, presentations, podcasts, and infographics with data-rich materials highlighting insights from the research. Research findings were recently highlighted in a SHRM feature article.
Versta Research was commissioned to survey American adults about attutides and behaviors around sun protection and skin cancer risk in both 2021 and 2022. Findings about sunscreen usage are featured in a recent WebMD news article; findings about skin cancer risks and misperceptions about sun safety have been featured in articles from U.S. News & World Report, NBC News, ABC News, Prevention Magazine, and other media outlets.
The Alzheimer’s Association commissioned research for its annual Alzheimer’s Disease Facts & Figures report, with a 2022 special focus on Mild Cognitive Impairment (MCI). There were two surveys: one of all U.S. adults, and another of U.S. caregivers. Survey results were highlighted in an NPR news segment, CNN, UPI, among other media outlets. An infographic of the survey findings is also available.
MORE VERSTA NEWSLETTERS