Home > Evaluation Methods > Cognitive Walkthrough

Cognitive Walkthrough

Description of method:

Cognitive walkthrough is a review technique where expert evaluators analyze or "walk through” each step of task performance in the documentation mentally. They act as if they are actually working through the task. Each step the maintenance technician would take is scrutinized.  This method finds mismatches between user's and writer's conceptualization of the task, poor wording choices, and inadequate feedback about the consequences of an action. 

Bottlenecks where the maintenance technician is blocked from completing the task indicate that the documentation is missing something – which might be clarification or another action that needed to be taken first.  Convoluted, indirect paths through task sequences indicate that it needs a revision that simplifies the task and collapses or restructures the sequence.  Areas which are confusing or unclear need to be noted to add information or change wording to clarify the meaning. This procedure uncovers implicit (unspoken) or explicit (clear) assumptions made by the writer about the user’s knowledge of the task.  The evaluation procedure takes the form of a series of questions asked about each step in the task that are derived from a theory of learning by exploration.

Development Lifecycle Stage:  This method can be used early in the document development to find areas that need improvement.  This may include language (grammar, typos), technical information (values, parts), illustrations (part identification, perspective), and procedural steps (sequence, necessity).

Number of Users/Evaluators Required:  The number of evaluators needed often depends on the complexity of the procedure to be evaluated. Less complex tasks may only require the attention of a single expert maintenance technician. More complex tasks will require four or five experts in order to adequately evaluate the procedure.

Type of Evaluators:  Cognitive walkthroughs require expert users. Since the actual parts are not  available, the expert must rely on memory and prior experience to detect and correct potential errors.  A mix of different evaluators is advised -- including the maintenance technician, maintenance engineer, design engineer, and field service engineer.  

Evaluator Skills required to Use the Method:  Although a basic knowledge of usability testing would be advantageous, this method can be successfully administered by reading about this method and using the cognitive walkthrough script to plan your evaluation.  Experts should have extensive knowledge of the procedure and systems involved. Evaluating maintenance procedures requires a detailed and global understanding of the spatial location of the affected part, adjacent parts, and structure. The expert will also need functional knowledge of the tools and processes required to complete the procedure. Finally, the expert should be aware of the potential effects the procedure will have on the system(s) as well as on related systems. (i.e. adding hydraulic power to system B without circuit breaker xyz pulled will cause main landing gear to retract).

Advantages of method: 

• The key advantages of this method is that it can identify problems very early in the development of the documentation development to find areas that need improvement.  This may include language (grammar, typos), technical information (values, parts), illustrations (part identification, perspective), and procedural steps (sequence, necessity).

• Helps define users’ goals and assumptions.

• This method uses fewer resources (time, personnel, and equipment) than actual user performance testing.

Disadvantages of Method: 

• The key disadvantage is that this method is more dependent upon evaluators’ biases and subjective comments.

• The errors and problems found using the cognitive walkthrough method of testing are less severe than those found in user testing.

• Experts are often forced to rely on memory of previous experiences. If the expert’s memory is incorrect or incomplete, potential errors may not be found or in some cases new errors may be introduced.

• It is often difficult for an expert evaluator to consider the maintenance procedure from the perspective of an inexperienced or naïve user.

• The process is most effective when the person conducting the evaluation is trained in cognitive psychology or the process of the cognitive walkthrough.  Untrained evaluators have been found to produce poorer results (Barnum, 2002). 

Level (or amount) of User and Evaluator Interaction:  There is little interaction between the person conducting the test and the evaluator reviewing the procedure.  The evaluator usually works alone while reading and thinking through the written procedure.  After the walkthrough, the person conducting the test will clarify the information that has been recorded on the procedure sheet to confirm their intended meaning. 

Data Recording Method(s):  Comments and corrections are recorded directly on the procedure.  Each participant should be given a copy of the procedure and any illustrations that are planned to accompany the written procedure.  Additional paper should also be provided for the evaluator for drawing or writing out their thoughts.  All comments are then combined for analysis and reporting.

Total Testing Time Required:  Depending on participants’ schedules, it is possible for the total testing being complete within one week.

Testing Time Per User:  Time required for each participant will typically range from 45 minutes to 1 ½ hours.

Typical Output from Test: The output from a cognitive walkthrough is subjective data, but is valuable in that it can be evaluated to make necessary corrections to the documentation and identify areas where the information is ambiguous.

How to Run the Test: This evaluation is usually conducted in an office setting, preferably in a quiet space with a desk. See Cognitive Walkthrough Script in the Tools & Templates section. Participants may be run simultaneously; however, there should be another person available to conduct the follow up interviews with enough distance as to not distract the other participant.

Required Testing Materials: The participants will need a copy of the written procedure, engineering blueprint, and any other supplemental information which will aid their evaluation.  Additional paper should also be provided for the evaluator for drawing or writing out their thoughts.

Cost to Conduct Test: The cost to conduct a Cognitive Walkthrough is relatively low. The cost can be estimated by taking the hourly salary of each experts times the number of hours required for review, including a group meeting if required.  Also, include your time in preparation for the review and recording the results.

References / Where to Learn More: See References and Useful Resources.

Type of System that Test Can Be Done On: Testing is most useful on complex procedures, but should be followed up with user performance evaluation after corrections are made.  Some questionable areas found in the Cognitive Walkthrough may be targeted for exploration during a user evaluation.

Goals of Testing: The goal of a Cognitive Walkthrough is to find errors in the text and illustrations at an early stage in the process of writing the procedural task.

Subjective or Objective Test: Cognitive Walkthrough evaluation is subjective from the frame of the expert reviewer’s reference; however, the expert evaluator is instructed to think through the procedure as a user unfamiliar with the procedure.

Ease of Learning to Conduct the Test: Conducting a Cognitive Walkthrough is relatively easy if there are experts in the necessary fields available to evaluate the procedure.  It is advisable to have a Usability professional as one of the experts to evaluate the documentation from a “user-centered” perspective.  Since the results do not require formal analyses, research experience is not required.

Turnaround Time: The time to produce deliverable results from this method is short if there is agreement between experts as to the proposed changes. Compilation and implementation of the findings are necessary to progress to the next stage of development, so it is necessary to come to consensus on changes quickly.

Focus of Evaluation: This type of evaluation has a limited or narrow focus in that only the documentation is available for review. 

Related Statistical Analysis: Results of the Cognitive Walkthrough are qualitative due to the low number of evaluators.