In quantitative survey research, the questionnaire affects nearly everything else that is critical to a study: the data, the analysis, and the findings. If the questionnaire is no good, all else is lost.

While not especially hard to write, questionnaires take surprisingly deep experience. They require thinking on multiple levels, even as one writes. You must ask yourself: How will respondents interpret and react to my words? How are they likely to respond or be swayed? How will I use their answers, and what kinds of analyses do I anticipate? How will I report the responses, and to whom?

Textbooks and methods courses in survey research teach you standard best practices for writing survey questions (use neutral language, avoid double-barreled questions, and so on). But beyond that, we find there are crucial “non-standard” best practices that researchers lose track of unless they have constant reminders or a checklist at the end.

There are crucial “non-standard” best practices that researchers lose track of unless they have constant reminders or a checklist at the end.

Here is our end-of-the-process checklist with helpful hints about the final last steps (five of them) we recommend in writing a questionnaire. These final run-throughs will ensure that you have given careful consideration to important issues in survey design that likely got pushed aside in early drafts as you focused on all those standard best practices in getting your questions right.

Review Your Don’t-Knows

A perennial debate among survey researchers is whether respondents should be allowed to answer “don’t know” or “not sure.” This is something we at Versta Research wrestle with (and argue about) internally almost every day. We have seen strong evidence in our own work that a large majority of don’t-know responses reflect an unwillingness of respondents, not an uncertainty. This is corroborated by a larger body of literature on survey research, as well (see our 2016 article: Don’t Know Is Not an Option).

Therefore we recommend against inserting don’t-know options unless absolutely needed. In fact, we suggest trying to write a questionnaire so that they are not needed at all. How? First, do not ask people for information they do not have. If “don’t know” seems like a reasonable response, write a better question to measure what you need. Second, look at your answer options. They should be exhaustive, comprehensive, mutually exclusive, and based on information from other research or expertise. If it seems that respondents will not find an answer that reflects their thinking, write better ones so that they will.

After you have done everything possible to revise your questions and answer options, run through the entire survey item-by-item once again to decide whether don’t-know is still a reasonable response, or just a lazy response. Add the option if it truly makes sense, but do so sparingly.

Review Your Other-Specify Boxes

It is tempting to add an open box at the end of every survey question so that respondents can write in their own answer if the pre-listed answers do not fit them. Our clients do this, and they ask us to do it, all the time. They are worried that we might miss something, and so we ought to let respondents tell us what we missed.

But there are big downsides to doing this. First, it makes data harder to work with. Every question with an other-specify box will  require manual review. You need to read what people wrote, and then decide what to do. Second, it adds ambiguity. Often people write answers similar to what you already supplied. Will you assume they were lazy, and change their answer to the one you supplied? Or will you assume they meant something different? Third, other-specify boxes do not work very well. If you decide to code responses into categories and report new answer options, the percentages will be low and probably not be worth reporting. And then those low percentages will likely be misleading because they represent unaided responses versus the others which were aided.

The biggest reason other-specify boxes do not work well, however, is that respondents will not use them. As much as we want them to help us, few will tell us what we missed. They assume it is our job to supply a comprehensive list of answer options (they’re right), and will answer as best they can within the constraints we offer.

So after your survey is drafted take a closer look at every question with an other-specify box. Keep it if it seems essential, but otherwise know that it will not deliver much in terms of “data” so you can safely let it go.

Review Your Randomization

Randomizing lists of response options is a good idea to even out the biased ways in which respondents search for answers to questions. Items at tops of lists are more likely to be selected, and items in long lists are likely to be overlooked. Why? Mostly because respondents read from top to bottom, search quickly for a good answer (even if not the best answer) and move onto the next question in the survey.

But randomization of response options is not always a best practice, so you should never randomize without careful thought. When you design answer options in Google Surveys, for example, you will be told: “Randomization produces best quality results.” This is not true. If answer options are a Likert scale, scrambling the order of the scale is a terrible idea. If answer options represent any logical progression (for example, local to global, general to specific, work-related to family-related) then randomizing creates confusion.

The same goes for whole sets of questions, especially items that are being rated in grid-like formats. Sometimes it is best to randomize, but not always. As you finalize your questionnaire, go through every set of questions and answer options and consider whether randomization is valuable to reduce likely biases. But also consider whether randomization will create confusion, and if it will, don’t do it.

Review Your Scales

Good researchers use a variety of scales, because every scale should be designed with specific needs in mind. Those needs vary from survey to survey, and from question to question. Sometimes we should use just two points (to boost reliability), sometimes four points (for PR-driven research), sometimes seven, nine, or eleven (for statistical modeling).

We always try to build such thinking into our drafts as we write survey questions and layout answer options. But it’s easy to lose track, and easier still to unthinkingly cut-and-paste scales from elsewhere in a survey. That’s why at the end, we review every question that uses ordinal scales. We ask ourselves: What are the minimum number of scales points we need for our analysis and reporting needs? Do we want to capture neutrality, or should we require respondents to lean one way or another? Should we offer a don’t-know or not-applicable option, or should we revise our questions and skip patterns instead?

Another piece to review is the order in which scales are laid out. Generally we want to list response options from low to high, or from bad to good. We start with things like terrible, zero, poor, disagree, and so on. Then we move towards spectacular, excellent, ten, and agree. This helps counteract a social desirability bias in which respondents lean towards giving answers that make them look favorable, pleasant, and positive.

Of course this is a general rule only! Sometimes negative responses are more socially desirable than positive ones. Sometimes flipping the sequence of response options causes confusion. So it is always essential to give careful thought, question by question, with focused attention on your scales, as you put the finishing touches on your questionnaire draft.

Review Your Skip Logic and Programming Instructions

On the very final review of a questionnaire, review skip logic and programming instructions in one focused pass, undistracted by the other issues you considered above. Good skip logic helps avoid the need for don’t-know response options, as you carefully adjudicate whether questions are applicable to every respondent. Good programming instructions ensure high quality data by further constraining (or not constraining) what respondents can see and do.

With skip logic, first confirm there is no ambiguity for analysts, data coders, and programmers by specifying every skip pattern in both words and programming language (example: IF WOMEN, Q2=2). Second, ask a colleague to proofread and imagine what the survey will be like for every type of respondent. If questions seem weird or inappropriate for some, add skip logic accordingly.

Then review and specify all programming instructions, and specify instructions for respondents as well. Review whether each question should be multiple response or a single response. In lists of multiple items, consider whether some should be exclusive, or always shown on the bottom (such as “none of the above”).

For open-end answer boxes, decide, specify, and explain constraints. If you ask for a number, can respondents enter text? Can they enter as many numbers as they want, and any range they want? Can they enter commas? We mostly advise against constraints, instead allowing respondents to enter whatever they want. It makes data cleaning harder, but it usually provides a better respondent experience, and has the added benefit of giving us more information to identify bad ones.