Testing

Materials

Needed:

1.      A copy of the procedure and accompanying illustrations

2.      Engineering blueprints and any other supplemental materials

3.      Stop watch to record the time each participant takes to evaluate the procedure

4.      Pen and writing pad

5.      Consent form (If you are including a video/computer simulation model during the test, be sure to include that information in the Consent Form.)

6.      Follow-up question sheet for recording responses

7.      Satisfaction questionnaire

8.      Background questionnaire

9.  (Optional) Any video or computer simulation model which has been developed to aid unfamiliar participants

Opening Statement:

(Meet the participant and escort into the room.)

“Before we begin I would like to thank you for agreeing to help. My name is _________________________ and I will be facilitating this session. Your participation in this study is very important. Your comments and insights will help _____________ develop a more usable maintenance manual. Before we begin I would like you to read over and sign a consent form. (Allow participants to keep a copy of the consent form if requested) Also, can you please fill out this background questionnaire? This information will help us better understand who you are.”

Handout

Consent form

Directions for testing session

“Before we begin, I would like to remind you of your right to withdraw from participation at any time. Should you choose to exercise this right; no repercussions or questions will be asked.”

“Your participation is also confidential.  Any answers or discussions from this interview will not be associated with your name.  We are collecting data from several participants and will evaluate the results collectively.” 

(If you are going to have the participant watch a video, etc., this should be done before giving the participant any further explanation regarding the actual Cognitive Walkthrough.) 

“During this session you will be given a maintenance task to read. I will not be able to answer any questions regarding task; however I can clarify wording. I would like to emphasize this is not a test -- we are evaluating the procedure, not you. Any questions so far?” 

“As you read through the task, pretend you are at work and try to imagine completing the task name for the first time as it is written.  Imagine each step you will need to take to accomplish the task.  As you “think through" the task name, please note directly on the procedure areas in which you would question the step as written. Imagine that the only information you have to complete this task is written in this procedure.   Try to write as much information as possible as to why you identified this as a problem.  There is additional paper to draw or write your information if you wish.”   

“Do you have any questions before we begin?” 

(Record the participant’s “begin” time on their Follow-up question form.)

Hand out

Give the participant the procedure / illustration(s). Have all supplemental materials available on the desk, including engineering blueprints, writing pad, and pen.

(When the participant is finished reviewing the procedure, record their “end” time on the Follow-up question form.

Hand out

2. Satisfaction Questionnaire

(To avoid any bias from comments by the evaluator, satisfaction questionnaires should be given before any discussion of the Cognitive Walkthrough evaluation.)

Follow-up questions

(When the participant is finished reviewing the procedure, record their “end” time on the Follow-up question form.  Go through their copy of the procedure with them to make sure you understand their notations.  Write any clarifying comments in a different color ink on the procedure.  Then sit with the participant and ask them a set of prepared questions, recording their responses on the sheet.  As an example …)

•  What strategies did you use to walk through the tasks?

•  How do you visualize the aircraft?

•  What is the most critical information you look at?

•  What supplemental information (if any) did you use and  where?

•  What other resources would have been helpful?

•  How confident are you that you found all of the problem areas?  (use a 1 (not that confident) through 5 (very confident) scale)

• How easy was it thinking through this task on paper versus viewing the actual aircraft?

Others…?

Handout

Satisfaction Questionnaire

Background Questionnaire

Closing Statement

Your comments and insights were very valuable to us. Once again I would like to thank you for your time and participation in our study. (Escort the participant out)