Scenario: Email Client Selection
Description:
Pat Brown, a decision-maker in a 3000-employee company is trying to decide
on a standard email client for the corporation to support. She
has been given documents recommending N-mail and U-mail, and must
decide which one better meets the company's requirements.
The authors of the two documents (John Smith and Jane Doe) are
available to collaborate with her to help make the decision. The goal
of the collaboration is to determine which email client to support,
and write up a jointly authored document explaining which client was
chosen and why.
The materials are skewed to make one package the "correct" choice.
The decision-maker is only one who knows the task. She must explain
it to the two others.
IC&V Program Objectives Addressed:
- enable real-time discovery of relevant collaborators and
information within task context,
- improve task-based performance of collaborators by two
orders of magnitude.
Domain Applicability:
- decision making
- summary document generation
Characteristics Tested:
[??]
Application Data and Format:
Scenario Data and Format:
[no environment-specific data is required]
Scenario-Level Evaluation Metrics:
Although each of these can be measured after the completion of a
particular sub-task, they reflect on the scenario as a whole...
Product Quality/Characteristics:
- Objective:
- Length of summary
- Number of requirements from original documents addressed
- Is the answer correct?
- Subjective:
Participants rate the following from Strongly Agree to
Strongly Disagree:
- the best option was chosen
- both options were fully explored
- the requirements were well understood
- the final summary was well-written
- etc...
Process Measures:
- number of requirements mentioned in discussion
- for each tool, how many requirements were compared to
capabilities
- comparison of the methods used in the CSTBE as compared with an
expert suggestion of the "best" way to do everything
Task Level Evaluation Metrics:
For each task:
- total elapsed (clock) time
- Unscripted version: proportion of time spent advancing task
vs. figuring out technology
- Scripted version: amount of time spent in unplanned divergence
from script
Tasks:
Setup: Learn Scenario Background
Set Up A Collaboration
Discuss and Agree on a Client
Prepare a Summary of the Decision