Home > Tools & Templates > Questionnaire Guidelines

Questionnaire Guidelines

You should apply the basic rules of user-centered design to the design of questionnaires.  Design a questionnaire in much the same way that you would develop a maintenance procedure (except that your design cycle might be in days rather than months).  

1.    You begin by gathering requirements that you need to know about users and the manual.  

2.    Follow the requirements gathering with a clear definition of goals and an explicit statement of how to build trust and provide respondents with benefits that outweigh the costs of filling out the questionnaire.

3.    Conduct prototype reviews and iterative testing.

4.    Ensure that you have a data analysis plan so that you understand what to do when all the data is collected.

Gathering Requirements and Questions

  • Interview key “stakeholders” about what they know and don’t know about maintenance technicians and how they use the manual.  Stakeholders include actual users, sales/marketing, training/ development, documentation, senior management, and technical support.

  • Distribute 3x5 cards and ask stakeholders from the different groups to write 1-3 questions that they would like to ask users. Avoid asking for too much here. This technique can be useful for getting insight into what issues are most important for different groups.

  • Conduct a short brainwriting session. Brainwriting is an extension of brainstorming where each person writes a question on a card and then passes it on to the next person who then reads the previous question and passes the card on to the next person who sees the first two questions and adds a third question. The premise is that seeing the questions of others will prompt additional relevant questions. This can be done in about fifteen minutes at team meetings and yield a large selection of questions.

  • Conduct a focus group to find out what issues are important to key user groups – they can be an excellent source of requirements for questionnaire design (see Interview/Focus Group evaluation method, p. 15, and Focus Group Guidelines, p. 68). The more open-ended nature of a focus group can provide input for more structured online or paper questionnaires.


Be Explicit on the Goals of Your Questionnaire

Too often, the goals of a questionnaire and each question on the questionnaire are not clear. Is the purpose of the questionnaire to gather information before or after a usability evaluation, gather requirements, or understand what difficulties the maintenance technicians generally have with the manuals?  Each question on a questionnaire should be related to a specific business goal and user experience issue.

Consider How to Establish Trust, Increase Rewards, and Reduce Social Costs for Respondents

You can design your questionnaire to create trust among respondents and influence the respondent’s expectations about the benefits and costs associated with filling out the questionnaire. Dillman (2000) notes that you can increase trust in the questionnaire by:

  • Providing tokens of appreciation in advance (though be careful not to make the tokens too large since this may be a source of bias)

  • Indicating clearly that the request is legitimate and the results can initiate changes

  • Making the questionnaire appear important

  • Indicating how the data will be used

Dillman’s (2000) suggestions for increasing rewards to respondents include:

  • Design an interesting questionnaire.

  • Use positive language that makes the respondent feel like a collaborator.

  • Provide tangible rewards.

  • Thank the user for helping.

  • Ask people for advice.

Suggestions for reducing the costs of completing a questionnaire include

  • Make the questionnaire usable.

  • Avoid embarrassing questions (don’t ask "how old are you?").

  • Minimize the need for personal information.

  • Make every question is relevant and avoid lengthy questionnaires.

  • Allow users to change answers easily in online surveys.

Create Prototypes of the Questionnaire and Review Against Principles of Survey Design

Design a prototype questionnaire, including the cover page, and compare it with the principles of questionnaire design. These principles should cover language, relevance, page layout, response categories, and ordering of the questions. I recommend that the questionnaire designer ask four people to review the questionnaire, and that you interview a few people not closely associated with the project as they read the questionnaire and think aloud about their reactions to it.

Devise a Data Analysis Plan

A common error in designing and implementing a questionnaire is to not devise a data analysis plan that spells out how answers will be coded.  For example, how will you code non-responses, unusual responses, or ratings where people circle two numbers when you only want a single answer.  You also need to consider what analyses you will do on single questions and sets of questions, any hypotheses that you may have and what questions will be used to test those hypotheses. You should do this even if you have survey software that does an automatic analysis of the data. You might find that your automated software doesn’t allow some of the analyses that you need to answer the questions that are important to your stakeholders.

Conduct Limited Testing of the Questionnaire with Actual Users

Get a small sample of users (or people as close to the expected users as possible) and have them fill out the questionnaire under realistic conditions and give you feedback.  Make your final changes based on this input and do a final edit.

Principles of Questionnaire Design

  • Ensure that your first question is relevant to everyone, easy, and interesting.

  • Avoid vague response quantifiers when precise quantifiers can be used.

In example 1a, the response categories are vague and can be interpreted differently by respondents. The data from this question would be nearly impossible to interpret. Example 1b eliminates the vague quantifiers with more specific answers.

1a. How often did you ???? during the last month? (Check 1 answer)
__ Never
__ Rarely
__ Occasionally
__ Regularly

1b. How often did you use ????  during the last month? (Check 1 answer)
__ Not at all
__ 1-3 times a month
__ Once a week
__ Two to four times a week
__ Once a day
__ More than once a day
Human Factors at NIAR | Human Factors at FAA | Human Factors Psychology at WSU

Human Factors Laboratory, National Institute for Aviation Research at Wichita State University. Research funded by the Federal Aviation Administration.  All rights reserved.
Revised: 11/05/04