Home > Evaluation Methods > Co-Discovery

Co-Discovery User Evaluation

Description of the method:

This method is designed to capture the user’s reactions and concepts when tackling a task for the first time and their experiences during their completion of the task.  This implies that we want the maintenance technicians to interact with the procedure with as few verbal instructions as possible.  To get the technicians to reveal what they think of and how they feel about the written procedure, pairs of subjects are asked to perform the task and answer questions together.  Dialogues and interactions may be video taped for analysis.

Co-Discovery may involve two maintenance technicians (one familiar with the procedure and one not familiar) or pairing a maintenance technician with an engineer.

Structure of a Co-Discovery session:

(adapted from Kemp & Van Gelderen, 1996)

Development Lifecycle Stage:  This method is most effective in the mid- to latter-stage of development -- when the aircraft is available and the documentation has already gone through internal proofing and heuristic evaluations or cognitive walkthroughs to detect errors. 

Number of Users Required:  Three to five pairs of maintenance technicians, or pairs of technicians and engineers are needed.  For a safety critical complex procedure, you might want to pair more experienced with less experienced technicians to find additional problem areas. 

Type of Users:  Most often, a diverse group of users is preferable.  The differences within the types of maintenance technicians will reveal different issues in the documentation.  The more diverse the input received, the more likely the documentation will be usable to a larger population.

Evaluator Skills required to Use the Method:  This method requires a great deal of creativity and skill from the evaluator; therefore, the best case would be an specialist in evaluative testing.  Many times the evaluator will not know exactly what areas will be problematic; therefore, the evaluator must be alert to the technicians’ actions, expressions, as well as dialogue.  This ability will aid the evaluator in probing the maintenance technicians’ thoughts and concerns during the session.

Number of Evaluators Required:  Depending upon the complexity of the procedure, one or two evaluators are necessary.  If the procedure is complex (number of steps or safety critical), two evaluators -- one to probe and one to video record -- may be advisable.

Advantages of method: 

• The maintenance technicians are in a relaxed state of mind during the evaluative sessions, and may enjoy participation.

• This method can improve understanding of how maintenance technicians both perceive and experience the documentation.

• The interactive nature of this method potentially produces many relevant issues and insights that may not be found in a single user evaluation.

• The video data is relatively easy for everyone to understand.  It is a straightforward way to communicate findings to others.

Disadvantages of Method: 

• It is a subjective evaluation and does not provide quantitative data of the documentation.

• Solutions to problems uncovered may not be straightforward to correct.

• It is more difficult to watch two people working with each other than one working alone.

• It is very time consuming to evaluate the video tapes.  This is dependent upon the number of people involved which prolongs the discussion periods.

• The costs for conducting this type of investigation is higher due to the number of maintenance technicians/other subjects required (two users vs. single user), the space requirements, aircraft access, video recording and analysis; however, it may save resources by correcting errors earlier in the documentation process.

Level (or amount) of User and Evaluator Interaction:

The level of interaction between the users is high; the interaction between users and evaluator is low to moderate.  The evaluator’s role is that of a passive facilitator and should only intervene in the discussion to probe or clarify information.   

Data Recording Method(s): 

Video taping the session is advised in order to have a record to re-evaluate your findings.  Portions of the video recording will also provide a clearer meaning when presenting the your findings to others. 

Note taking by the evaluator is also helpful but somewhat difficult due to the flexible nature of interactive probing; however, data may also be recorded by a skilled note taker.  Portions of the test may need further clarification following the evaluation and the note taker can mark those areas for probing in the debriefing session.

Total Testing Time Required:

This testing method will take weeks to months to arrange schedules of maintenance technicians/others, the aircraft, and recording/videotaping, in addition to the actual testing process. 

Testing Time Per User:

Depending upon the complexity of the task, Co-Discovery evaluation will take approximately 50% longer than the procedure would "normally" take when using participants unfamiliar with the task.  This additional time is due to using a think-aloud protocol, having others observe the user's performance, and the debriefing session following the evaluation.  Less time would be required for the experienced user.

Typical Output from Test:

Collected data will be both objective and subjective.  Objective data can be collected for Co-Discovery user evaluations, such as the time users take a complete the task, the time spent recovering from errors, the number of errors discovered, how frequently the manual was used to solve the user's problem or the number of times the user had to workaround a problem that wasn't covered in the procedure.

Subjective data can also be collected as to the participants' comments during the evaluation about the language/terminology used, the sequence of the tasks, the participants' comments during the debriefing session following the performance evaluation, and the satisfaction data collected by the questionnaire.

In addition to the data collected during the session, the video/recording may also reveal problem areas that are not seen during the actual testing.  Viewing the video after each session is advised in order to further explore problem areas in subsequent sessions.

Related Tests:  Co-Discovery is basically the same as User Performance Evaluation using two participants rather than a single user.

How to Run the Test:  Basic instructions for running the tests can be found in the Tools and Templates  section -- the Usability Evaluation Planning Template  to help with each of these stages – the User Testing Script, User Performance Evaluation Guidelines for Facilitator and Note Takers , Background Questionnaire, and Satisfaction Questionnaire to collect individual user's information.  The basic instructions are the same for Co-Discovery and single user performance testing  .

Required Testing Materials:  Audio/video equipment, aircraft or component parts with any needed tools, the written procedure with any illustrations or supporting documentation, a consent form, background questionnaire, and a satisfaction questionnaire.

Cost to Conduct Test:  High due to the human (two participants) and aircraft resources required.  The cost may be reduced by using component parts on the bench rather than the aircraft.

References / Where to Learn More:   See References and Useful Resources.

Type of System that Test Can Be Done On:  User testing can be performed on the actual system or component parts on the bench.

Goals of Testing:  A Co-Discovery evaluation finds the most severe problems in the documentation -- the issues that usually are not discovered using other methods.  Therefore, the goal of this type of user testing is to discover problems in the maintenance documentation for the most complex and/or safety critical components of the aircraft.

Subjective or Objective Test:  The results generated from Co-Discovery evaluations is mainly objective but may be biased by the evaluator's perception of the participants' comments.  Care should be taken when an the evaluator is not trained in user testing that the participant's comments are clarified when in doubt.

Following the data collected from the evaluation, a satisfaction questionnaire will collect the participant's subjective assessment of the procedure.

Ease of Learning to Conduct the Test:   In the case of evaluating technical maintenance documentation, user evaluation is principally testing the documentation as written and discovering areas that require more, less, or different text and/or illustrations.  Therefore, the ability to learn how to conduct this type of test requires a moderate level of reading and no formal research training.  A basic understanding of cognitive processes, the user's action cycle, and careful planning of the User Evaluation are necessary to get valid and reliable results.  There are many written resources available to help as well.

Another consideration is the potential for bias if the same person conducting the test has produced the documentation being tested.  When possible it is best to have another source conduct the testing with the video tape available for viewing by the documentation's writer.

Turnaround Time: User testing using the Co-Discovery method allows for results to be used along the way.  By seeing the different approaches to problem areas by various technicians/engineers, the writer will have the ability to probe for solutions, test alternatives, and/or add necessary information for clarity where required.  Depending upon the availability of resources necessary to conduct the user testing, turnaround for the completed project may take several weeks.

Focus of Evaluation:  The focus of Co-Discovery evaluation is wide since the users have access to the test article and the full extent of the maintenance task can be evaluated.

Related Statistical Analysis: Results of the Co-Discovery user evaluation are qualitative due to the low number of participants.