Last year we offered a two-week mini-course to undergraduate business marketing students at the University of Illinois in Chicago. The first week consisted of lectures with Q&A, covering the basics of real-world market research and “making research matter.” (Curious? You can watch our 2-minute introductory video on Making Research Matter.)

For the second week, students formed teams and completed a mini-research project of their own. They had to conceptualize, design, execute, analyze, and present it to the class. Yikes—they had never done this before, and they had just one week to finish!

They had never done this before, and they had just one week to finish!

At the end of the mini-course, we asked them to provide written comments answering two questions:

1. What was the biggest surprise about research they learned, especially from the lecture?
2. What was the hardest part of doing a research project on their own?

With pages of comments from more than sixty Gen Z students in hand, we read them all to find common themes, tallying the most frequently mentioned surprises and challenges.

The Nine Biggest Surprises

1. Lay out findings before writing survey questions. A guiding principle for us—and hugely helpful technique—is to have clients write out dream headlines before we jump into questionnaire design. What’s the ideal outcome or number they want to report to their manager? What story would they share with the media, if only we could find that in our data? Students were surprised that beginning with the end in mind means conceptualizing the findings first.
2. Dream headlines do not mean confirmation bias. One can lay out hunches and dreams of what the data might say (that is what specifying a hypothesis is all about!) and still avoid confirmation bias. How? By writing unbiased questions, inviting respondents to tell you the opposite, and building a search for “disconfirming evidence” into the research protocols. Good researchers want to know (and are excited by) the truth, even if it runs counter to what they expected to find.
3. There is always a good story in the data. Even if dream headlines or ideal outcomes are not supported by one’s data, good research design—driven by dream headlines—pretty much guarantees a good story. This is because it helps you write smart questions relevant to the topic, and truthful answers to those questions matter. We have never had a project fail or end with a shrug, even those that go against initial expectations.
4. Words are more powerful than numbers. For reasons we do not fully understand, saying “one-third” has more narrative impact than saying “33%.” Therefore we encourage colleagues (and students) to lean towards fractional words and nouns or adjectives like most or few in their reports, using numbers for data-specific reference. Example: Nearly all students (95%) who attended the mini-course completed their homework assignment on time.
5. Data are not the story. When summarizing and reporting, start out with a narrative that does not use statistics or numbers. It helps keep focus on what the data mean, which is the story you want to tell. Then bring in data afterwards as supporting documentation. Data and statistics are never the story; they are simply the proof that the story is true. Same goes for charts and graphs. They should never be the focal point; they should be the proof you show that the story is true.
6. Data tell many different stories. Depending on how it is framed, data can be used to tell multiple, even “opposite” stories. A large majority (83%) of our U.S. Olympic athletes are vaccinated. What good role models for our communities! Wait, you mean nearly one in five (17%) are not vaccinated—and yet they’re travelling to Tokyo, in contact with other athletes? What is wrong with them?! A skillful researcher can frame the data to tell whatever relevant, compelling story managers find more useful.
7. Use percentages and fractions, not averages. One of the first statistical calculations we learn in school is the mean, or average. But it is tough for people (your audience) to think in averages. Instead, they think in terms of clusters, fractions, and groups of people—for example, half, most, fewer than one in ten, and so on. As such, we rarely report the mean score on survey scales. It is better to report a percentage of respondents who used one end of the scale or the other.
8. Numbers are just numbers. One of the dumbest gimmicks newspapers and magazine sometimes use is the formulaic “by the numbers” layout. The headline might be a number, like: 83. Then the supporting copy would say, “The percentage of U.S. athletes who are vaccinated.” But honestly, do you think “83” is interesting? We hope not. It’s just a random number. Even in research, numbers are just numbers, so always keep focused on the story that give those numbers meaning and interest.
9. Clients want pretty charts. Students were surprised when we told them that clients value the artistic and design-minded side of what we do. But of course they should! Well-designed charts (with all superfluous decorations removed) are pleasing to look at because they convey data in clear, succinct ways that focus on the story that needs to be told. Good researchers should be involved in chart design just as they must lead the initial survey design.

The Six Biggest Challenges

1. Dreaming up Headlines. This was hard for students because, unlike real research, they made up their own projects. It can be hard for researchers, too, but it doesn’t need to be. Just ask your client or the internal team who wants the research to do it for you! Almost always, they have an agenda and a dream. Ask them to specify it. Have them write five to seven dream story headlines of what they hope the research will find.
2. Specifying the Data Needed. Once the students had headlines, they needed to work backwards to topics, questions, and measurement scales. At every point they needed to ask, “How will I report this data when finished, and how specifically will it support the narrative I am likely to tell?” We agree, this is one of the biggest challenges of good research, and it will continue to be for their entire careers. It takes lots of experience to know how every data decision is assessed in terms of how it will get reported.
3.  Writing Good Questions. It was good to hear that students found this one hard. Because it is, indeed, hard if one is focused on the data that will be generated, how it will be reported, and how it will support the findings laid out in dream headlines. And that’s just half the battle. Writing unbiased questions, designed to elicit disconfirming evidence is hard, too. Ideally, these marketing students will never do what most marketing professionals do when they want research: jump into writing survey questions.
4. Finding Survey Respondents. You think finding survey respondents is hard? Well, join the club, we say! Your client (in this case the professor) gave you just one week to do it? Welcome to our world! Getting real people (not fraudsters), who meet your screening criteria for inclusion to offer heart-felt opinions about topics that may be of only marginal interest to them is always one of the most challenging aspects of market research. Research panels help, but they do not solve the problem.
5.  Assembling Data into a Story. Indeed, this is hard too, but it is fun. It’s like an art project that begins with assembling the materials, sifting for the beautiful nuggets that speak to (but do not always confirm) the headlines laid out when you started, distilling each to its essence, pulling them into meaningful wholes, and helping the narrative take shape. If this is something you need to do (and struggle with) yourself, take a look at our “how-to” guide on Turning Data Into Stories®.
6. Creating Visuals to Support the Story. This was hard for students because they relied on Microsoft’s default graphics and chart layouts, not quite knowing how to improve them with good design. Choosing effective charts, tables, and infographics, is an important element of doing good research, and it cannot be handed off to programmers at software companies. It requires careful attention to the story, to the nature of the data, and to the comparisons that need to be highlighted.

The overall lesson these marketing students seemed to have learned from our mini-course—evidenced by what they found surprising and hard—is exactly what we hoped for: Doing great research is challenging, but not because math and statistics are hard. It is because great research requires a wide perspective and set of skills that go far beyond math. It requires a deep understanding of people, psychology, narrative, design, science, and logistics.

Doing great research is challenging, but not because math and statistics are hard.

We expect the students in this course will probably become marketing professionals, not researchers. And we hope they remember how tough this assignment was, and will deeply appreciate those researchers in their future who provide them the data, the stories, and the strategic answers they need.