Home > Evaluation Methods > Questionnaires

Questionnaires

 Description of the method:

Questionnaires are written lists of questions that you distribute to your users.  Questionnaires differ from surveys in that they are written lists, not personal interviews.  You begin by formulating questions about the documentation based on the type of information you want to know. You will need to anticipate every question you might ask and put them in written form.  The questions also need to be put into the most specific and appropriate form.  There are examples of Satisfaction and Background Questionnaires, as well as Questionnaire Guidelines in the Tools & Templates section.

 Development Lifecycle Stage: 

This technique can be used at any stage of development, depending on the questions that are asked in the questionnaire. Often, questionnaires are used to measure user satisfaction or to assess background information about the participant in an evaluative study.  In this case, satisfaction or background questionnaires may be useful in any stage of development when the testing is done. The questionnaire can also be used to collect user or task information that will be used early in the manual’s development. 

Number of Users Required: 

Background and satisfaction questionnaires should be administered to all participants in user performance evaluations.  When using questionnaires to collect information for task analysis, you should try to get at least ten responses for each user group for which you are interested in gaining information.

Type of Users:  

The user groups will depend upon the research question, but may include novice and/or experts, maintenance technicians, technical support personnel, and engineers.

Evaluator Skills required to Use the Method: 

Questionnaire development requires knowledge of creating effective questions in the most appropriate form.  There are several excellent resources available to learn this skill including:

Dillman, D. (1985).  Mail and telephone surveys:  The total design method.  New York:  John Wiley and Sons. 

Payne, S. L. (1979).  The art of asking questions.  Princeton, NJ:  Princeton University Press.

Sudman, M., & Bradbum. A. (1982).  Asking questions:  A practical guide to questionnaire design.  San Francisco:  Jossey-Bass.

Alreck, P. L., & Settle, R. B. (1994).  The survey research handbook : Guidelines and strategies for conducting a survey.  New York:  McGraw-Hill.

Advantages of method: 

• Finds subjective user preferences.

• Gains background information of user groups to compare.

• Adds valuable data without much additional cost.

• Can get information from large numbers of people, so can include many representative groups.

• Can ask many questions.

• Data can be collected rapidly.

• Responders remain anonymous.

Disadvantages of Method: 

• No flexibility in gathering data; all participants receive the same questions.

• There is no way to know if the participant understood the question in the way in which it was intended.

• Indirect method of data collection; cannot be sure that the person intended to provide the information is the person who filled out the information.

• Cannot obtain in-depth thoughts

• Does not obtain information about the context of the situation.

• People's responses may not accurately reflect what they actually do on the job.

• Response rates may be low which can reduce generalizability of results.

(Johnassen, Tessmer, & Hannum, 1999; Dumas & Redish, 1994)

Level (or amount) of User and Evaluator Interaction:

There is very little or no interaction between the user and evaluator. 

Total Testing Time Required:

Depending upon the number of questions asked, usually questionnaires require no longer than 15 minutes to complete.  Generally, satisfaction and background questionnaires take less than 5 minutes to complete.

Typical Output from Test:

Output from questionnaires is self-reported subjective or preference data.

How to Run the Test:

Questionnaire guidelines as well as examples of a Satisfaction and Background questionnaire are included in the Tools and Templates section.

Related Tests: 

Questionnaires are similar to surveys in that both are a written list of questions.  The difference is the participant fills out the questionnaire in written form rather than responding the questions verbally in a survey.

References / Where to Learn More:

See References and Useful Resources.

Cost to Conduct Test:

Low costs to develop, print, and administer questionnaires.

Goals of Testing:

The goal of questionnaire evaluation is to administer the same questions to different users to assess differences and similarities in their subjective opinions between or among groups of users.

Subjective or Objective Test:

From the perspective of the individual user the results of a questionnaire are subjective; however, when results from large groups of users are averaged together, the results are an objective measure of satisfaction or preference.

Turnaround Time:

If the questionnaire is administered as a part of another evaluation, such as user performance or co-discovery, a questionnaire is given and received without delay. 

In some instances, a questionnaire may be developed to collect other user information.  In this case, there will likely be a delay if it is sent by mail or an internal method.  It is best to give a “due date” as well as a turn key means to return the questionnaire.

Focus of Evaluation:

The questionnaire method can be used to collect many different types of user information with either a wide or narrow focus.  Most often it is used to collect specific user background information and post task/post test to collect satisfaction or debriefing data.