The DreamPad evaluation will consist of the following techniques: * Cognitive Walk-Through * Heuristic Evaluation * ClassRoom Simulation Evaluation Exercise 1 --------------------- Cognitive Walk-Through (Theoretical) ------------------------------------------------------------------------------- A cognitive walk-through will be conducted on both the student and instructor portions of the DreamPad Application. The evaluator will run the application according to the user's guide and answer the following four questions for each of the actions performed: 1. For each goal listed (0,A, B, C...), do you expect to take action Ai Bi Ci...respectively? Student Application: * 0. Connect to the teacher o 0.1 Enter IP address in edit box I found this slightly confusing because the application showed a default IP address. At first, I tried to connect with that IP address and later realized that the IP address was wrong. If you expect students to enter IP address, you should show a blank window initially. o 0.2 Press Connect button I expect to take this action. * A. Ask the instructor a question o A1. Click the question Icon or Press the Ask->Question menu item I expect to take this action. o A2. Type your question into the edit box I expect to take this action. o A3. Press Send Button. I expect to take this action. * B. Give opinion of Class Mood To give opinion on class mood, which action should I perform? Should I set the mood meter or set the signal color? o B1. Click the thermometer icon + B1.1 Move the scroll bar to the desired position I expect to take this action. + B1.2 Press the Send Button I expect to take this action. o B2. Move to the Mood Meter Menu + B2.1 Move the selection to the correct setting I expect to take this action. + B2.2 Select the item I expect to take this action. * C. Give Feedback on a particular slide If you want feedback on a particular slide and expect a student to vote only once, i. Tell the student that he can vote only once. ii. Turn the appropriate button off when student sends a vote. In the present system it is not clear whether a student is allowed to send multiple votes. However,assuming that a student can send multiple votes the following answers, o C1. Click on the thermometer icon + C1.1 Click on the desired light I expect to take this action. + C1.2 Click on the send button I expect to take this action. o C2. Move to the StopLight Menu + C2.1 Move the selection to the correct setting I expect to take this action. + C2.2 Select the item I expect to take this action. Instructor Application: * A. Get Student Questions o A1. Observe there are questions queued I expect to take this action. o A2. Press the "?" icon I expect to take this action. o A3. Read each question I expect to take this action. o A4. Press OK when done I expect to take this action. * B. Read the class mood Is it necessary for the instructor to click the meter icon? If I am an instructor, I will expect that the icon itself show the mood. Only if I want to see it enlarged should I click on the icon. o B1. Click the meter icon However, once I click on the meter icon, understanding the reading is clear. o B2. Read the text/Observe the meter reading o B3. Click OK when done If a teacher clicks on the class mood icon and forgets to close the resulting enlarged window, the meter should get updated as the students give further feedback. * C. Read student input on current slide Here you are showing the number of votes as well as the color of the signal. The teacher naturally starts wondering what algo is used to determine the colors. Maybe, you can tell the teacher just the number of votes and let him determine the feedback. o C1. Observe and Interpret Stoplight Colors o C2. Observe and Interpret Text Display o C3. Observe and Interpret Vote totals * D. Advance/Retreat a Slide o D1. Press the appropriate Button I expect to take this action. * E. Interpret Post-Class Results This action is not evident on running the application. May be you can provide a feedback that you are saving the questions after each slide. o E1. Launch NotePad or some other text reader o E2. Open the appropriate "qq" file. Why not use a better name? o E3. Interpret Data 2. At any particular stage, is there anything (within the range of goal A, B, C...) that you want to do, but cannot find the correct action? I can list the following, A. Student : 1. I wanted to change the IP address after I said OK to the default IP address. There is no provision for this. 2. Can I send multiple votes on the class-mood? If not, there should be a message telling that when I try to send second vote. B. Teacher : The teacher wants to see the question with particular number. There is no way for the teacher to do that. 3. Is the result from the action expected by you when you take the action? A. Student: 1. When sending a question, the ENTER key takes you to the next line. One may press ENTER to send a question. B. Teacher : 4. Do you understand the feedback of every action? 1. When I send a question or any other kind of response, there is no feedback. The window just disappears. I think there should be a feedback after the question is sent. 2. You can constantly show the IP address at the bottom to show that the student is connected . Evaluation Exercise 3 ---------------------- Heuristic Evaluation: (Optional) ------------------------------------------------------------------------------- Instructor Dialog: * A. Visibility of System Status o Does the Instructor Display provide adequate feedback to determine: + That a question has been asked? + How many questions are pending? + How well the students understand the current slide? + What slide the instructor is on? The instructor has to remember how many questions have been asked if he wants to know immediately the arrival of a new question. There should be a feedback on how many questions have been asked and how many have been addressed. ( Something similar to mail software ). Also there should be some way for the teacher to go to the current slide if he goes back to some other slide while presenting this slide. o Does the Student Display provide adequate feedback on student selections? The student should be have a copy of questions he sends. * B. Match between System and Real World o Do the displays use terms familiar in the classroom? Yes. o Is the information displayed in a logical order? A student can indicate negative feedback in two ways. ( Using the mood meter and turning on the RED light ). This makes the two indicators slightly redundant. o Is the stoplight paradigm understandable? Yes. * C. User Control and Freedom o Do all student dialogs provide a cancel function to retreat from selection before send? Yes. o Was there any time you felt you had inadequate control over the application? 1. I want to see what questions I have asked before. 2. I want to see if someone else has already asked a question before I ask a question. 3. I have no feedback that the question I asked has reached the teacher properly. o Were you able to activate the application dialogs when you wanted? Yes. * D. Consistency and Standards o Do button functions operate within normal MS Windows parameters? Yes. o Are there inconsistencies between DreamPad controls and those an experienced Windows user would expect? While asking a question, pressing ENTER should send the question. In the present system it takes you to the next line in the editor. Also, it will be better if a student is given option of setting the editor for typing question. Can it be made something similar to sending mail while browsing through netscape? o Are the operations of dialog boxes consistent throughout the application? Yes. o Were you ever confused by any command or feedback? At some instances, there is no feedback. * E. Error Prevention o Could erroneous input cause the application to fail? The application fails in the following cases: 1. Entering wrong IP address ( There is no way for the user to know whether he is properly connected) o If the application failed, what were the circumstances? 1. Typing wrong IP address. * F. Recognition over Recall o Were all user options visible? Yes. o Could all input be achieved without entering commands from memory? Yes. o Were prompts and icons understandable? Yes. * G. Flexibility and Ease of Use o Were adequate accelerators provided for expert users? Not applicable. o Were the command's purpose obvious? Yes. * F. Aesthetic and Minimalist Design o Were dialogs clear and uncluttered? o Do you think any part of the design was redundant? o Are any design elements "ugly" or unattractive? o Was there extraneous information on the displays? o Were the displays too large or too small? o Was needed information not shown? I have the following general comment in this regard: 1. Redundant design: The moodmeter and the light signals both give similar kind of feedback. If a student is really bored he may send negative feedback using both or may even get confused as to which form of feedback he should send. The application appears redundant in this regard. 2. Design elements are good and definitely serve the purpose. 3. No. 4. The size appears ok. The user should have option to resize. 5. When the instructor changes a slide it is saved to a disk file. There should be a feedback here or the save message should be shown. Similarly, feedback should be provided for all other actions. * H. Help Users Recognize, Diagnose, and Recover from errors o Did you make any mistakes or slips during use? Yes. 1. Typing the wrong IP address. o If so, what was the system feedback, and was it adequate? There was no feedback. o If error messages were encountered, were they understandable? I suppose there are no error messages in the present version. I did not come across any. o Were the troubleshooting instructions in the user's guide adequate? Yes. * I. Help and Documentation o Was the users guide adequate? Yes. o Was the information grouped for easy understanding? Yes. o Were the dialog labels adequate? Yes. Evaluation Exercise 3 --------------------- Classroom Simulation (Participatory) ----------------------------------- Description of the exercise: In order to have an idea whether the application complements the classroom interaction and provides a useful tool, it was necessary to use the DreamPad application in conjunction with CLASSPAD during an actual live class. This could have given a better idea of the usability of the DreamPad in an actual classroom setting. This however could not be done because the DTR devices networking port is not currently working. So, a classroom test was simulated with a user population of 5 who played the role of students. One idea for selecting a user population of 5 is that, according to Nielsen,a user population of 5 is sufficient to discover 75% of the overall usability problems. The development team however does not specify any particular number for it. They have specified just a "group of users" to evaluate the system. One of us, as an evaluator, played the role of the instructor and attempted to present a lecture using a slide format to the users. A short orientation of the DreamPad was conducted and all the users were actually given an opportunity to get themselves acquainted with system before the actual simulation was carried out. Results: During simulation, the evaluator of our team, as the instructor, collected the following data: 1. Does the instructor use the question feature to retrieve student questions? Ans. As an instructor, the evaluator used the question feature to retrieve the student questions. But there are several problems with the interface in the way retrieving questions is designed. Firstly, the questions are not numbered. So there is no way to keep track of which question it is. Secondly, there is no way to go to the previous questions. So the teacher has to go through each question in a FIFO order. If there are similar questions, there is no way for the teacher to go to previous questions to compare them. So, there is chance of time loss, in an attempt to answer similar questions asked by different students, unless the teacher keeps track of the answered questions Also there is no facility available with this system to assign the slide number for which the question was asked. So, there is inconvenience associated in the sense that the teacher manually has to figure out to which slide the question is actually related or has to go to qq file. 2. Does the instructor use the mood indicator to determine the class mood? If so, is the information valuable? How does he use the information presented? How does he interpret the information shown? Ans.The evaluator, as an instructor did use the mood indicator to determine the mood of the class. The inconvenience associated with the design of the moodmeter is that the evaluator had to manually go and click the moodmeter in order to determine the mood of the simulated class. It would have been better if the interface were so designed as to let the instructor see the average mood of the class continuously, like the stop light icon (traffic light). As a result the instructor does not get continuous feedback on the average mood of the class and how often it is changing. Only when (s)he clicks it, (s)he gets the feedback. In order for the instructor to get proper feedback, (s)he has to press the meter at regular intervals which is not a good design. So, the information presented by the mood meter is not very valuable. Also it gives just the average mood of the class. Since it is not always feasible to continuously check the meter, it is not possible to get feedback for each individual topic that is covered in class ( the evaluator tried to talk about different topics in simulate classroom). So, it is not always possible for the instructor to get enough feedback for each individual topic i.e if the topic is exciting or boring. So, the use of mood meter is somewhat redundant particularly in the in-phase part of the class. Another reason for it to be redundant is that it serves the same purpose as the stop light icon. The only use of the moodmeter for the teacher is in the post-class phase where at the end of the class (s)he gets an idea of the average mood of the class. So, this information is usually of no use for the current students. Whenever the instructor happens to click on the moodmeter, depending on that particular feedback can change lecture pattern. But this is also not correct. While lecturing on a particular slide, if (s)he finds that the reading of the meter indicates boredom, (s)he may speed up on that particular slide. But this might be the feedback the students gave for the previous slide( which the instructor did not see) and had not given any feedback at all for the present slide. So, rather than improving the interaction, the moodmeter can actually make the interaction between the teacher and the students worse. 3. Does the instructor use the stoplight icon to determine how well the slide is understood? If so, is the information valuable? How does he interpret the information presented? Ans.The stoplight icon serves a better purpose than the moodmeter. The instructor gets constant feedback from this icon for each individual slide. The icon,to some extent, gives the instructor information about how well a slide is understood. The algorithm does not support history and number of people in class. If for a particular slide, only five people press green, the instructor can move to the next slide fast. But the rest of the class might not even understand what was going on with the slide before (s)he can press any button. So, for this undecided part of the class, the actual status of the class is typically not reflected by the stop light icon. Only five choices have been selected by the development team randomly. So, the stop light information is valuable only when all the students are actually participating in class i.e there is no student who is undecided. Depending on which light appears on the instructor's console, the instructor slows down, presents the slide in more details or speeds up. Also the instructor has to remember exactly which light stands for which action since no such feedback is provided in his/her console. So, if (s)he forgets what green stands for there is no way for him/her to know that and this may lead him to take a wrong action. If only a single light per slide appears on his/her console, (s)he can take take necessary actions nd this gives him/her some idea about the degree of complexity for each slide. Also if the counter value is the same for all the three lights, only the red appears on his/her console leaving him/her at his/her own discretion as to which action to take. 4. Have the instructor review the question file after class. Does the information presented provide adequate feedback for him to adjust the slides or modify the presentation in the future? Ans. One of us who played the role of the instructor did review the questions asked, after the class. The problem with design of the question icon is that the questions are not numbered and they don t carry the information as to which slides they refer. Also, there is no provision for the evaluator to go back and look at a previous questions. All these constraints make the modification of the slides pertaining to a lecture quite difficult. But from the different questions asked, it was possible for the evaluator to adjust parts of the slides and make them more explicit for better understanding of the students and helped him/her decide whether to spend more time on a particular slide. These modifications could lead to better presentation of material from a particular slide. During evaluation, we, as the evaluators got the following feedback from the users who played the role of students. The following presents an average input of all the 5 users. An actual interview with a user who played the role of the student is attached in the appendix. Question icon: The "student" used the question icon to send question to the teacher. The positive part as stated by most of the users is that it lets them ask question to the user anonymously. But there are many negative sides of this. When one raises his/her hand in class, one gets the question answered immediately, it is appropriate for the student and also for the rest of the class. But typically an instructor chooses to answer the question at the end of the class. The question being answered after such a long time (after so many topics and slides covered in class) invariably defeats the purpose of the question being asked and the student most of the time loses interest. So, in that respect it is no better than the old way. There is no way by which the student can see what questions others have asked. So there is high possibility of questions getting repeated and there is no way to eliminate the repeated questions by the instructor beforehand. This results in wastage of time. There is no way a student can remove a question from the list. The student cannot send multiple questions at the same time. The student also gets no feedback by the system whether his/her question has reached the instructor. There was a button with slide typed,but the users had no idea what purpose it served. Stoplight icon: The student used stopligt to signal his/her understanding for a particular slide. This enabled them to convey their overall understanding in the class. If more students vote for a particular slide to be explained in more detail, the instructor can do that. Also they can request the instructor to go over a slide slowly or fast depending on the complexity of it. Thus on and average the stoplight serves the purpose. But a worse problem occurs when a student is undecided. In that case only minority of the students can vote for an action and can actually have the instructor go with it. The system does not maintain the history of each student's understanding and let the minority rule in these instances. Moodmeter: The students in general did not communicate their mood to the instructor. They rather preferred to ask questions and send messages to stop or speedup or explain a slide in more detail. According to them, these were the actions that eventually got responded. These actions were also precisely defined and let the instructor take some relevant decisions. The moodmeter only conveyed the average mood of the class which really did not let the instructor to take a clearcut action. Also this may be misleading in many cases. Since the instructor had to manually click the meter, more often than not, he preferred the feedback in form of questions and stoplight. So, he would infrequently check the meter. But the feedback the meter gave might not be relevant to that particular slide. So, the feedback for a previous slide could prompt the user to take a wrong decision regarding current slide. Also the students said that the meter was more of a use for the instructor for delivery of his/her future lectures rather than the present class. So, some users were even reluctant to use the meter. 5. The bottom line is really, do the participants actually use the DreamPad tool as a compliment to the classroom experience, or is it mearly a novelty? Ans. The DreamPad does improve the interaction between the students and the teacher. The students can anonymously ask questions without interrupting the class. Also this gets rid of the students not asking questions because of peer pressure, cultural norms or intimidation. The instructor also gets to answer the questions at his/her convenient time without actually getting interrupted. The students can express their feeling for each slide and can actually have the teacher go through the slides slowly if they are difficult for them or have the teacher just gloss over the slides that appear to be boring. This definitely improves their understanding for each slide. The teacher can have a better understanding of the level of complexity of each slide which (s)he can improve upon for future lecture. But there are drawbacks too. The teacher has to go every now and then to check the moodmeter to determine mood of the class and correct his action accordingly. This results in some wastage of his/her lecture time. Also, this can give him/her wrong information for a particular slide and lead him/her take wrong action. For example, he may check the meter at the time of a slide when no feedback was at all given by the students and take an action of speed up or slow down based on the feedback of some previous slide(s) or has to go to qq file. This has a negative effect so far the students are concerned. The students are also affected in the sense that some may be undecided regarding a particular slide but the teacher takes an action based on whatever color appears on his/her console. This also has a negative effect on the student. Though getting questions answered during class itself is good for the students or slowing the instructor down for a slide improves their understanding, there is much time wasted in the instructor checking the moodmeter, the questions pending and answering them, going back to each individual slide again and again (since one never knows about the next question, one can see it in FIFO order only and the questions do not carry information as to which slides they refer to). With all these considerations, the DreamPad is more of a novelty than it is of actual use in class.