Home > Tools & Templates > Usability Evaluation Plan

 

Usability Evaluation Plan for (Project Name)

The following is a template for planning a User Performance Evaluation.  Each element of the plan is explained and then supported with an example. 

Purpose of the Usability Testing
Defining the purpose of the user performance evaluation is perhaps the most important step of this entire process.  The purpose consists of two factors – the objectives and the goals.

This section specifically lays out what, why, and how you intend to use a usability study to accomplish your goals.  By “thinking through” what, why, and how when planning your testing, you will assure that you are actually answering the question(s) you intended to and gain additional insights into your project.

Objectives:

• Some specific concerns to be addressed in this user performance evaluation:  (This section outlines the specific concerns of the writer.)

For example: 

The following objectives will be addressed in this usability evaluation:

• Is there adequate information in the written procedural task for the user/maintenance technician to install/remove/modify the (component)?

• Will the user’s/maintenance technician’s performance be improved by including an illustration with the written task?

• Is the sequence in which the information is ordered efficient for the user/maintenance technician to perform the task?

User Performance Evaluation Goals:

• Specific user performance goals are developed from the objectives outlined above.  The goals let us create evaluation scenarios and tasks that test the validity of our concerns and develop measures to use in user testing.  The objectives listed above should be translated into measurable performance goals for testing. 

For example:

This evaluation will be based on the following user performance goals:

• The user/maintenance technician will be able to install/remove/modify the (component) from the written task.

• Maintenance technicians will be able to perform the task with no additional illustration.

• The user/maintenance technician will be able to accomplish the task in the sequence written with no visible difficulty.

• Maintenance technicians will be able to find related information with no expressed or visible frustration.

• Maintenance technicians will be able to complete tasks within 150% of the benchmark times.

We will also use a satisfaction questionnaire to determine subjective reactions, such as:

• Maintenance technicians feel that the task is arranged in an efficient sequence.

• Maintenance technicians feel that the task is written to aid understanding of the technician.

• Maintenance technicians feel that the task includes all relevant information needed to complete the task.

Target Audience Evaluation:

• Maintenance technicians should be selected to provide an adequate sample of the group that will be using the manual; therefore, the criterion should be set as to the technician’s background that represent all types of maintenance manual users. 

• Describe the background of all maintenance technicians, including pertinent information such as education, experience level (# of years, # of types of aircraft, etc.), skills, job functions.  The critical differences between technicians should be determined before the participants can be selected for testing.

Selection of Participants:

• Participants should be selected to reflect the range of characteristics determined in the target audience evaluation.  The types of maintenance technician characteristics needed will be dependent upon the type of task and aircraft to be evaluated.

For example: 

The following levels of the maintenance technician will be testing the usability evaluation:

• Education Level - High school, Certifications, College degree

• Years of Experience - Three levels:  0-3; 4-7; or 8 or more 

• Types of Aircraft:  Two levels:  3 or fewer types; 4 or more types; or Two levels:  Commercial or Private 

• Skills/Job Functions - (Dependent upon the writer’s project.)

Design of the User Performance Test:

A single user performance evaluation should be run on at least five (5) participants in individual sessions.  Each session will consist of a performance evaluation, e.g. giving the maintenance technician a set of procedures (or task) to complete, a follow-up interview, a satisfaction questionnaire to collect the maintenance technician’s subjective comments about task procedure, and a background questionnaire.

Evaluation Process:

1)     Introduction

• The participants will receive a short, verbal scripted introduction to the evaluation.  An example of a User Test Script can be found on page ?. This material will explain the purpose and objective of the evaluation.  This will also explain the advantage the usability testing will have on their job efficiency.

• They will be assured that the manual is the center of the evaluation and not themselves; and they should perform the procedures in the manner that is typical for them.

• The evaluator will let the participant(s) know that the session will be held in the strictest confidence.  Their individual performance and opinions will be pooled with others and not looked at separately.

• The participant(s) will be told the general purpose of the evaluation and the advantage to their job efficiency.

• The participant(s) will be asked to sign a Consent Form. 

2)     Testing

There are three roles for individuals involved in the User Performance Evaluation.  Each evaluation role should be identified in the Evaluation Plan.  The maintenance technician as  the "Participant"; the evaluator as the “Facilitator”; and the “Note Taker”.  (See Facilitator and Note Taker Evaluation Guidelines)

The User Performance Evaluation is given in the following sequence:

• Introduction and sign consent form. See an example of a User Performance Evaluation Script.

•  

• The participant(s) will be given a written procedure to perform on the aircraft or component parts on a bench.

• Participants will be instructed to use a "think aloud" protocol method to record their actions and thoughts.

• Participant(s) will be video/audio recorded during the session to have a record for further analysis.

• Participants will be asked for further explanation when a problem area is incurred.  After the task performance, the Facilitator will clarify comments made during the evaluation from the Note Taker's record.  The further clarified comments will also be recorded.

• A satisfaction questionnaire will be given to the participant collect the maintenance technician’s subjective evaluation of the procedure following the task evaluation. 

• Each participant will be asked to fill out a short background questionnaire to collect information about the technician’s education, experience level (# of years, # of types of aircraft, etc.), skills, job functions as set out in the Selection of Participants section above. 

3)  Analyzing the Data and Reporting the Results:

    Following all of the User Evaluations, the data will be analyzed on the following:

• Identify participant errors and difficulties

• Analyze the source of errors and difficulties – technical, procedural, grammatical, or graphical.

• Identify what the corrective action should be of the errors and/or difficulties

• Prioritize problems by criticality – unusable, severe, moderate irritant, etc.

• Analyze differences between participant experience levels