Evaluation Plan

The DreamPad evaluation will consist of the following techniques:

• Cognitive Walk-Through
• Heuristic Evaluation
• ClassRoom Simulation

Cognitive Walk-Through (Theoretical)

A cognitive walk-through will be conducted on both the student and instructor portions of the DreamPad Application.

The evaluator will run the application according to the user's guide and answer the following four questions for each of the actions performed:

1. For each goal listed (0,A, B, C...), do you expect to take action Ai Bi Ci...respectively?
2. At any particular stage, is there anything (within the range of goal A, B, C...) that you want to do, but cannot find the correct action?
3. Is the result from the action expected by you when you take the action?
4. Do you understand the feedback of every action?

Student Application:

• 0. Connect to the teacher
  • 0.1 Enter IP address in edit box
  • 0.2 Press Connect button
• A. Ask the instructor a question
  • A1. Click the question Icon or Press the Ask->Question menu item
  • A2. Type your question into the edit box
  • A3. Press Send Button.
• B. Give opinion of Class Mood
  • B1. Click the thermometer icon
    • B1.1 Move the scroll bar to the desired position
    • B1.2 Press the Send Button
  • B2. Move to the Mood Meter Menu
    • B2.1 Move the selection to the correct setting
    • B2.2 Select the item
• C. Give Feedback on a particular slide
  • C1. Click on the thermometer icon
    • C1.1 Click on the desired light
    • C1.2 Click on the send button
  • C2. Move to the StopLight Menu
    • C2.1 Move the selection to the correct setting
    • C2.2 Select the item

Instructor Application:

• A. Get Student Questions
  • A1. Observe there are questions queued
  • A2. Press the "?" icon
  • A3. Read each question
  • A4. Press OK when done
• B. Read the class mood
  • B1. Click the meter icon
  • B2. Read the text/Observe the meter reading
  • B3. Click OK when done
• C. Read student input on current slide
  • C1. Observe and Interpret Stoplight Colors
  • C2. Observe and Interpret Text Display
  • C3. Observe and Interpret Vote totals
• D. Advance/Retreat a Slide
  • D1. Press the appropriate Button
• E. Interpret Post-Class Results
  • E1. Launch NotePad or some other text reader
  • E2. Open the appropriate "qq" file.
  • E3. Interpret Data

Heuristic Evaluation: (Optional)

Instructor Dialog:

• A. Visibility of System Status
  • Does the Instructor Display provide adequate feedback to determine:
    • That a question has been asked?
    • How many questions are pending?
    • How well the students understand the current slide?
    • What slide the instructor is on?
  • Does the Student Display provide adequate feedback on student selections?
• B. Match between System and Real World
  • Do the displays use terms familiar in the classroom?
  • Is the information displayed in a logical order?
  • Is the stoplight paradigm understandable?
• C. User Control and Freedom
  • Do all student dialogs provide a cancel function to retreat from selection before send?
  • Was there any time you felt you had inadequate control over the application?
  • Were you able to activate the application dialogs when you wanted?
• D. Consistency and Standards
  • Do button functions operate within normal MS Windows parameters?
  • Are there inconsistencies between DreamPad controls and those an experienced Windows user would expect?
  • Are the operations of dialog boxes consistent throughout the application?
  • Were you ever confused by any command or feedback?
• E. Error Prevention
  • Could erroneous input cause the application to fail?
  • If the application failed, what were the circumstances?
• F. Recognition over Recall
  • Were all user options visible?
  • Could all input be achieved without entering commands from memory?
  • Were prompts and icons understandable?
• G. Flexibility and Ease of Use
  • Were adequate accelerators provided for expert users?
  • Were the command's purpose obvious?
• F. Aesthetic and Minimalist Design
  • Were dialogs clear and uncluttered?
  • Do you think any part of the design was redundant?
  • Are any design elements "ugly" or unattractive?
  • Was there extraneous information on the displays?
  • Were the displays too large or too small?
  • Was needed information not shown?
• H. Help Users Recognize, Diagnose, and Recover from errors
  • Did you make any mistakes or slips during use?
  • If so, what was the system feedback, and was it adequate?
  • If error messages were encountered, were they understandable?
  • Were the troubleshooting instructions in the user's guide adequate?
• I. Help and Documentation
  • Was the users guide adequate?
  • Was the information grouped for easy understanding?
  • Were the dialog labels adequate?

Classroom Simulation (Paticipatory):

While the two previous tests are likely to be sufficient to determine the usability of DreamPad, it fails to adequately address a critical question, does the application complement the classroom interaction and provide a useful tool. The purpose of the classroom simulation is to provide not only another indication of the usablity of DreamPad in an actual setting, but to provide an indication of its degree of usefulness in the classroom. The ideal test of course would be to use DreamPad in conjunction with CLASSPAD during an actual live class. Unfortunately, the DTR devices networking port is not currently working, so the classroom test will have to be simulated.

For this test, you will need at least 2 PC's running Windows and linked together via a network. One PC should run the instructor console and the others the student console. An instructor (or evaluator playing the role of an instructor) should attempt to present a "lecture" using a slide format to a group of students (who may be other evaluators). If using actual users, a short orientation to DreamPad should be conducted prior to the exercise. During this simulation, the evaluator's should collect the following data:

Instructor:

• The instructor should be observed for his use of DreamPad.
  • Does the instructor use the question feature to retrieve student questions?
  • Does the instructor use the mood indicator to determine the class mood? If so, is the information valuable? How does he use the information presented? How does he interpret the information shown?
  • Does the instructor use the stoplight icon to determine how well the slide is understood? If so, is the information valuable? How does he interpret the information presented?
• Have the instructor review the question file after class. Does the information presented provide adequate feedback for him to adjust the slides or modify the presentation in the future?

Student:

• Does the student use the interface to ask questions? If so, does he feel it is a "better" way than the old way?
• Does the student use the stoplight to signal his understanding of a particular slide? If so, does he feel this is of benefit to his overall understanding of the class?
• Does the student communicate his mood to the instructor? If so, does he feel this helped in improving the class?

The bottom line is really, do the participants actually use the DreamPad tool as a compliment to the classroom experience, or is it mearly a novelty?

Back to the Design Document