In this paper, we describe efforts to develop and evaluate a large-scale experiment in ubiquitous computing applied to education. Specifically, we are concerned with the general problem of capturing a rich, multimedia experience, and providing useful access into the record of the experience by automatically integrating the various streams of captured information. We describe the Classroom 2000 project and two years of experience developing and using automated tools for the capture, integration and access to support university lecture courses. We will report on observed use of the system by both teachers and learners and how those observations have influenced and will influence the development of a capture, integration and access system for everyday use.

Ubiquitous computing, educational application, capture, integration and access of multimedia

The interest in ubiquitous computing has surged over the past few years, thanks to some influential writings [17, 18] and plenty of experimental work. A defining characteristic of ubiquitous computing is the attempt to break away from the traditional desktop interaction paradigms and move computational power into the environment that surrounds the user.

One of the potential features of a ubiquitous computing environment is that it could be used to record our everyday experiences and make that record available for later use. Indeed, much of our everyday lives is spent listening to and recording, more or less accurately, the events that surround us, only to have that one important piece of information elude us when we most need it.

There is a value to using computational resources to augment the inefficiency of human record-taking, especially when there are multiple streams of related information generated by a group of people. It is virtually impossible for a human to record all aspects of such a rich interactive experience. Computational support can also automate the relationships between various streams of information, so that one stream of information, such as notes taken during a meeting, can be used to index into another stream, such as the audio track from the same meeting. In this way, an integrated, multimedia record of an experience can aid in the recall of the meaning or significance of past events.

We have just outlined the general capture, integration and access problem for ubiquitous computing [2]. Tools to automate the support for capture, integration and access can replace an activity humans are not good at (e.g., recording) with other, more suitable tasks (e.g., summarizing, and interpreting).

As Weiser points out, "Applications are of course the whole point of ubiquitous computing.'' [18] It is not enough to identify the general problem of capture, integration and access for ubiquitous computing. We are compelled to explore and understand it better through specific applications. Over the past two years, we have examined this problem within the context of university education, in a project called Classroom 2000 [1]. The objective of the Classroom 2000 project is to create an environment to capture as much of the university classroom experience as possible. We have developed tools to automate the production of on-line lecture notes that integrate the various streams of information common to the classroom experience.

We have built a special-purpose classroom and iteratively developed a suite of tools. Over the past 18 months, our systems have been used regularly in a dozen undergraduate and graduate courses. This paper reports on our observations of how a capture, integration and access system is used, how it impacts teaching and learning practices, and further issues on how the system has evolved over time and how we see it evolving in the future. Ultimately, we want to use our experience with Classroom 2000 to:

• Demonstrate to researchers in ubiquitous computing the importance of engineering robust systems that can be the basis of large-scale and long-term studies; and
• Inform educators on how ubiquitous computing technology can be used to improve both the teaching and learning experience.

Our work has been greatly influenced by previous research at Xerox PARC in ubiquitous computing [17,18] and tools to support electronic capture and access of collaborative activities [4,11,12,13]. We want to provide the students with a way to capture more personal notes through some form of an electronic notebook. The Marquee note-taking prototype developed at PARC [16], the Filochat prototype developed at Hewlett-Packard Labs [19], and the Dynomite personal note-taking environment from FX-PAL [20] come close to what we wanted to have in the hands of the students. Another possibility is to augment the capabilities of a traditional paper notebook, as done by Stifelman [15]. We have experimented with personal notebooks, but our most significant experience has been support for public display surfaces in the classroom using a large-scale electronic whiteboard like the Liveboard [4].

When it is not practical to produce such a transcript, we rely on the implicit relationship between an audio stream and time-stamped pen strokes or keystrokes. This time-based relationship directly supports the task of a user browsing a set of notes that was taken during a class and asking "What was the lecturer talking about when this was being written?" This form of integration is exploited by the note-taking prototypes mentioned above, as well as other work at MIT's Media Lab [7] and at Apple [3].

The assumption in all of the preceding work in this area was that only one person will ever access the recorded notes and associated audio or video streams. Our work, in contrast, assumes that captured information in the classroom supports a number of people, bringing up issues of sharing public records and preserving the security of private notes.

There is a very large and deep body of literature in educational technology. It is not our intent to review that field, but it is important to place the problem of capture, integration and access within the larger context of educational technology.

A lot of the effort to introduce technology in the classroom equates to placing a workstation in front of each student. This is appropriate for some classes, but we want to investigate the usefulness of alternative pen-based interfaces that are amenable to the presentation and handwritten note-taking activities that occur in most classrooms.

We consciously decided to support the lecturing and note-taking activities that already occur in the majority of university classes. Some will complain that we are only trying to use new technology to improve a bad way of teaching and learning. An important part of our research methodology is to put a system into use quickly and with a variety of users. Only through long-term use will we be able to observe the impact of our system on teaching and learning. It is much more practical to provide an initial system, therefore, that requires relatively little change in the practice of the users. Over time, we fully expect that our systems, if useful, will evolve to be used in ways that we did not and cannot predict.

A problem with educational software is content generation. Years of effort can go into producing a single course module that represents only a fraction of a student's educational experience. Every year, however, universities provide thousands of hours of instruction in a myriad of subjects. By viewing the classroom experience as a multimedia authoring session, we simplify content generation. Though our current efforts are to support students who actually attend the classes that are captured, it is certainly feasible that over time a number of captured classes can themselves be packaged in a way to augment future educational objectives. We leverage heavily off of the World Wide Web as a delivery mechanism for this content. This more active use of the Web infrastructure is in tune with some recent educational technology work [10,14,5].

Proper research in educational technology has a foundation in some theory of learning that is backed by empirical results. In our case, we are focussed on the learner in a typical lecturing situation. A very common task for the learner is to take notes and to use those notes as a study guide [8]. Kiewra et al. [9] explain that note-taking is a complex task which allows little time for generative processing (active learning) to occur, and that review of the notes allows for more deeper understanding and reflection. A recent study [6] of the relationship between note-taking, review and student ability showed that students with high working memory capacity performed better in classes when they primarily listened during lectures and reviewed notes provided by the lecturer. These theories on note-taking and the somewhat surprising result lend credence to our intuitions. Providing better ways to capture the classroom experience can improve learning if students are encouraged to listen and engage in the lecture with confidence that notes will be provided afterwards.

Several of the research prototypes cited above have been subjected to some form of evaluation to determine both usability (do users find the system simple to use for the purpose it was designed) and usefulness (does the system enhance the user's ability to perform some task). The two most substantial evaluation studies have been conducted at PARC and Hewlett-Packard. For a two-year period at PARC, a suite of tools was used to support a process of intellectual property management [12,13]. At Hewlett-Packard, the Filochat system was evaluated in field and laboratory studies, generating both qualitative information on the reaction of users to the technology and also quantitative information comparing the accuracy, efficiency and confidence of Filochat with paper-based and dictaphone systems [19].

The evaluation in this paper is largely qualitative based on 18 months use of various prototypes in actual courses. To date, we have used our systems in a dozen courses within the undergraduate and graduate Mathematics and Computer Science curriculum at Georgia Tech. During that time, we have administered questionnaires to students and teachers, tracked student usage of the system (both anonymously and with student identification) conducted ethnographic style interviews with some students, and observed videotapes of lectures. All of this data collection has been used to help us modify the capabilities of the system based on user requests and to help us better understand over time how the system is being used and why. It is unwise to conduct quantitative experiments to gauge the impact on specific learning outcomes prior to having such an understanding.

Fielding a ubiquitous computing system that is robust enough for everyday use in a classroom environment is a very challenging prospect. The engineering issues can easily outweigh the research challenges. Initially, the effort involved to support a single class was so great that we were only able to experiment with one class per quarter. Over time, we were able to streamline all features of the system and support 5 classes during the Spring quarter of 1997. We have identified four separate phases of the system execution and built or bought tools to support those phases. The phases are briefly described below. Interested readers can explore more of the Classroom 2000 tools via our Web site (http://www.cc.gatech.edu/fce/c2000).

Before a lecture begins, some lecturers prepare a presentation as a series of slides or even Web pages. In this pre-production phase, we are very concerned with providing a simple way to reuse existing classroom material. Our electronic whiteboard supports the annotation of GIF images. We modified some public domain UNIX scripts to convert Postscript output to a series of GIF images that can then be loaded into our electronic whiteboard system. Recently released commercial applications, such as PowerPoint 97, have built-in conversion utilities that also allow for the creation of high-quality GIF images from presentation slides.

Once a lecture starts, we want to record as much information as possible. Since most teachers use some sort of blackboard or whiteboard during a lecture, we provide an electronic whiteboard to capture what is presented and written. We have written several whiteboard programs for pen-based computers, the latest of which is called ZenPad, a client-server Java system. The front-end client of ZenPad, shown in Figure 1, runs as a Java applet inside a Web browser. An earlier Visual Basic prototype whiteboard, ClassPad is discussed in [1]. ZenPad provides a very minimal interface that maximizes the actual screen surface for writing. Earlier prototypes provided more functionality to the end user, but those features were not used very much and subtracted from the all-important screen real estate teachers wanted for a writing surface.

Figure 1: The ZenPad electronic whiteboard. On the left, we show prepared slides over which the teacher can write. On the right, ZenPad is just a blank whiteboard that continually scrolls upward to provide more writing space.

Our electronic whiteboards provide two features that were lacking in commercial counterparts we investigated. First, the artifacts that are produced are easily distributed via the Web. The ZenPad whiteboard can also be used in a browse mode for reviewing lecture notes after class and, for the large number of students lacking a robust Java-enabled browser, a pure HTML version of the notes is also provided. Second, our electronic whiteboards keep information about when various activities occur. For example, we know when a lecture began, when slides were displayed and when every single penstroke was created. This information is critical for the integration of the teacher's lecture material with the audio or other streams of information that are recorded during class.

The teachers we supported had lecturing styles that required two different presentation styles. Teachers with prepared slides wanted to have ZenPad move slide by slide through the lecture with the ability to display a slide and annotate on top of the slide. Other teachers simply wanted ZenPad to provide a continuous scrollable blank writing surface upon which the entire lecture could be handwritten. Figure 1 shows examples of ZenPad being used for both of these modes. Upon invoking ZenPad, the teacher had to identify which style was preferred for that lecture.

The audio track for the lecture is recorded both to a high fidelity DAT tape for later analysis and backup and to a lower fidelity digital streaming format for later delivery over the Web.

Once class is complete, there is some work necessary to provide for the integration between the captured electronic whiteboard activity and the audio stream of the class. One of the critical concerns here is the granularity of integration between handwritten notes and the audio stream, which we discuss later. There is also some post-production work necessary to produce a Web-based interface that makes it easy for the students to access the augmented notes from a class home page. This Web-based interface is the subject of the next phase.

In the access phase, students and teachers are provided a Web interface to browse past lectures. As we mentioned before, the ZenPad system provides a browse-only mode that is suitable for this purpose, but we have also produced a number of HTML interfaces. Figure 2 shows two examples of interfaces we have used. In all cases, there is the ability to jump from the displayed lecture material to the audio for that lecture. Different prototypes provided different indexing capabilities. For example, the interface on the left of Figure 2 allowed the student to index into the audio at time points defined by when slides were visited. In browse mode, ZenPad allows indexing down to every penstroke. The interface on the right side of Figure 2 provides a granularity roughly at the word level.

One of our major goals is to demonstrate by example that applications research in ubiquitous computing must proceed by putting large-scale systems into everyday use so that their impact can be observed. We have been very successful at engineering Classroom 2000 so that it can operate in support of a large number of classes and we can begin to observe how it is used in practice. This section will summarize our observations on the use and impact of Classroom 2000.

The observations discussed in this section are the result of a number of different evaluation techniques. Student reactions to Classroom 2000 have been obtained through post-course questionnaires, videotaped interviews, and access logs to course material. Teacher reactions were obtained via interviews, questionnaires and analysis of videotaped lectures.

Figure 3: Student requests for lecture notes. Data covers 5 courses taught in Spring 1997.	Figure 4: Student requests for audio integrated with lecture notes for the same 5 courses.

Analysis of Web logs to class material is somewhat tricky, since course notes were usually publicly available. To provide better quality data without sacrificing anonymity of the students, we restricted our analysis to IP addresses within our university sub-domain and eliminated obvious non-student addresses. We were interested in seeing how often students accessed the lecture notes. Five courses were supported in the Spring Quarter of 1997. Three undergraduate courses in Calculus, Artificial Intelligence, Software Engineering had average enrollment of 45 students and 2 graduate courses in Computer Vision and Machine Learning had average enrollment of 15. 89% of all students attempted to access the on-line lecture materials, with 78% reporting that they were successful in accessing the notes and 49% reporting success in accessing audio. One entire class received inadequate training for accessing notes, so our results are negatively skewed. As we had suspected, access to lecture notes showed dramatic peaks in the few days preceding exams (usually a midterm in the fourth week of term and/or a final exam during week 10 or 11). Figure 3 shows the number of requests for lecture material by week for all five classes over the quarter.

Figure 4 shows student requests for audio. The profile is similar to the general Web access, but we observe that audio requests are approximately an order of magnitude less frequent.

When asked for reasons why they accessed the on-line lecture notes, student responses were (out of 165 respondents):

• 27% to catch up on a missed lecture
• 14% to answer questions from own written notes
• 14% to help in doing homework assignments
• 36% to study for exams
• 9% other

70% of the students did not feel that the availability of captured lecture materials encouraged them to miss class. 78% felt that the technology in Classroom 2000 made the classroom experience more engaging or interesting. 58% responded that their note-taking practices changed during the quarter. 17 students explicitly stated that they stopped taking notes because of the availability of the notes after class and 30 stated that they could spend more time listening to and understanding the lecture instead of writing down notes. When asked if their study habits changed, 18 students said they had stopped relying on their own notes in favor of the captured notes. Despite this favorable reaction to the availability of captured lecture notes, 70% of the students did not feel strongly that their performance in the class improved as a result.

65% of the students felt that the technology in the classroom did change the way the instructor taught and only 8% thought that this change was for the worse. But how did the teachers react to the automatic capture of their lectures? We observed videotapes of 5 different teachers and surveyed the teachers to get some of their general impressions.

The teachers had one of two lecturing styles when using the electronic whiteboard. Two lectured with prepared slides and three simply used a blank whiteboard. One of the teachers would have liked to use prepared slides on occasion, but felt that it involved too much effort to incorporate into ZenPad.

All of the lecturers felt that the capture of their lecture materials was a valuable service that outweighed the negative features of the ZenPad system, described below.

All of the teachers agree that the size of the whiteboard was too small. The electronic whiteboard has a 67-inch diagonal screen that is suitable for use in a small conference room but at least one-third the size of a traditional whiteboard in a classroom that seats 50 students. Teachers felt unable to write as much on the electronic whiteboard as desired in an effort to conserve screen real estate. Several of the lecturers were tied to the electronic whiteboard and very much limited their movement and student interaction to a small portion of the classroom. A larger surface might remove that tendency.

Other negative reactions were to the overall speed and lack of naturalness to the electronic whiteboard. Loading pages would take anywhere from 2-5 seconds. This caused awkward pauses during a lecture and discouraged navigation back and forth through the lecture. Erasing was clumsy enough that most teachers preferred instead to cross out mistakes. Some of the teachers felt that a larger electronic whiteboard that displayed the recent history of a lecture would alleviate some of their frustration with navigation. Despite this overall clumsiness of the interface, we only noticed one teacher actually falling back to the regular whiteboard in the room.

Three of the teachers regularly used the electronic whiteboard to run programs other than ZenPad. The most common use was to pull up a Web browser. Though they were pleased to be able to do this, they were disappointed when that part of the class was not captured for students to see later.

We were interested in the relationship between what a lecturer said and what was written on the electronic whiteboard. Since timestamps on the penstrokes were the basis for integrating the notes and audio, it matters how synchronized those two streams actually are. There was no general trend to report across all teachers, but individuals were fairly consistent across lectures in how their writing synchronized with what they said. Two of the teachers (both teaching mathematical subjects) would speak as they wrote, so individual penstrokes were fairly closely aligned with the audio stream. One lecturer consistently verbalized an idea and then summarized it on the electronic whiteboard, usually without saying a word. Others would write something and then talk about it afterwards. The middle strategy will cause an index to overshoot the relevant audio, causing possible confusion for the listener.

The most experienced user of ZenPad was also the one whose lecturing style was most modified. This teacher said that he felt much more like a scribe on behalf of the class, and would frequently write marks on the board to assist students during the access phase. He even told the students that he was doing this to assist them. This teacher also had the practice of writing down student questions on the board, a practice he admitted he never did before using ZenPad.

Teachers said they were conscious that they were being recorded. While they did not feel this changed the way they taught, two noted that they were more reserved with their language.

In addition to gaining a better insight into how students and teachers have come to use our technology, we have identified some important issues for further consideration.

The electronic whiteboard must provide a larger interactive surface. As a result, we have built a prototype system that partners the highly interactive surface of the electronic whiteboard with a number of additional projected displays. We intend to use this system soon to determine if it alleviates any of the problems of the smaller system. Though our solution will provide a larger persistent display of lecture notes, the lack of interactivity may still be a source of frustration.

We have spent a lot of time automating post-production activities at the expense of attending to the start-up activities for a class. Our solution has been to provide human support at the beginning of lectures. This problem is exacerbated by our desire to record more and more streams of information in the class, all with appropriate backup. Every new stream of information captured (e.g., video, other programs) requires another human to kick-start the recording to ensure that all streams are properly synchronized. There must be a better way!

Despite this synchronization problem, it is very desirable to capture additional streams of activity in the classroom. One immediate goal is to use video in addition to audio. Teachers quickly started to take advantage of the large screen networked computer to pull up information on the Web during class or to demonstrate other software. This activity is not currently captured in the ZenPad framework. This past summer, we ran a Web browser displayed on a separate screen and had it produce a time-stamped log of URLs visited. We have also experimented with capturing arbitrary activity on one machine in the form of a QuickTime video. In addition to capturing more streams that will be delivered back during the access phase, there is also the need to capture streams that will be used to assist in the integration, as we will explain below.

There are three issues with integration. The first issue has to do with the granularity of time-based integration. What kind of control should we provide for linking lecture material to the audio stream? Our initial prototype, shown on the left side of Figure 2, offered resolution to an individual slide. Every visit to a slide would create another index into the audio stream. We felt this was too coarse, so in ZenPad a link to the audio stream is created for every penstroke (defined as a pen down to pen up sequence). Our experience has shown that this strategy is too fine. The interface shown on the right of Figure 2 uses a simple time and space clustering heuristic to group related penstrokes together. In addition, we have provided an interactive timeline that allows access to the audio without relying on a penstroke. This latter strategy is especially useful for lecturers who don't write that much.

Our integration scheme has been driven by a single stream, the teacher's writing. Other streams of information can also be used to provide more content-based integration. For example, we would like to use keywords to search the audio stream. We have built a prototype system that converts a high-quality audio stream to a searchable sequence of phonemes. We will use this to decorate a lecture timeline with occurrences of teacher-supplied keywords. We can also provide arbitrary, student-generated keyword searches within and across lectures.

Other streams can also be analyzed to support content-based integration. We can analyze video to detect gestures that associate what is going on in the class with some material being presented. It is often important to know what the focus of the class is at any point in time. The behavior of the teacher can be analyzed to determine which of a number of streams is the focus of discussion. Students themselves can produce actions which index into the classroom experience. Imagine a simple switch that a student controls that would indicate important points during the lecture. We have built such a "switch" using a Pilot PDA as the input device. We intend to experiment with these alternative integration strategies in the near future and report on their effectiveness.

One of the most important requirements of Classroom 2000 is that it produces universally accessible lecture material. An elaborate integration mechanism to facilitate content-based search is useless if students cannot execute it when and where they study. If students don't use a feature because they cannot get it to work, then our observations on use are tainted. Many of our system design compromises directly reflect this requirement. For example, all multimedia playback facilities must be cross-platform with reasonable performance over a 28.8 Kbaud modem connection.

This same requirement has caused compromises in the programming language we use to present information to the student reviewer. We use the Web as the delivery infrastructure, and we want to use Java as a cross-platform language to develop highly interactive access capabilities. Unfortunately, commercial browsers that support Java do not do a very good job yet, so we have had to relax our desire to use Java exclusively. Our most reliable access interface consists of inline GIF images with client-side image maps written in HTML. Our long-term strategy lies beyond the capabilities of HTML, as we will demonstrate.

Several students requested a playback feature in which the audio actually drives an animation of the lecturer's notes. This feature would make it easier to follow a lecture while listening to the audio. In general, we need to provide better ways to visualize the entire contents of a stream to facilitate searching. Being able to quickly scroll through thumbnail images can help to locate a section of the lecture of interest.

We have experimented with students taking their own personal notes with a tablet-sized electronic notebook. Some simple controlled experiments reported in [1] indicated that the individual notebooks were ineffective, so we stopped. It is still a goal, however, to allow students to produce private notes that are integrated with the public notes created by the lecturer. We want to provide a way to protect a student's privacy while also allowing them to share their personal notes with others as they see fit.

It is a serious restriction that captured notes are immutable. Students often want to review a lecture and write summaries or otherwise modify their notes taken during class. For one class, we provided an interface that allowed students to edit their own private copy of notes from a lecture. We sacrificed this feature when scaling up to support multiple classes, but we will soon re-introduce it.

We are well on our way to satisfying the research goal of demonstrating how to do applications research in ubiquitous computing. There are further issues of scale that we must still address. As Classroom 2000 expands from a single room to an entire campus and from isolated classes to an entire curriculum, we will face grand challenges in system design and information retrieval.

As our understanding of the actual use of capture, integration and access in education increases, we will be able to address our other goal related to assessing the impact on teaching and learning practices. In the coming years, we will be able to conduct long-term studies to measure the effectiveness of our approach and observe how teaching and learning practices evolve to suit the capabilities of ubiquitous computing technology.

We have defined the general problem of capture, integration and access for ubiquitous computing and demonstrated its application in a large-scale educational application to support university lecturing. The goal of Classroom 2000 is to determine what positive impact ubiquitous computing can have on teaching and learning. Before we can make any solid assessment of the new technology, however, we must first understand how it is used in large-scale, everyday practice. This presents significant engineering challenges that any ubiquitous computing researcher must face.

We have succeeded in implementing an environment that provides a reasonable amount of automation to record a classroom experience. Eighteen months of everyday use has fed an iterative development cycle. Our observations show that students have come to depend on captured lecture material and have reduced their in-class note-taking and increased the effort to listen and participate in the classroom experience. With this favorable result, we are encouraged to begin deeper assessment studies of the impact of capture, integration and access on teaching and learning objectives.

Johan LeMon and Tommy Enqvist are visiting students to the GVU Center from the Royal Institute of Technology in Stockholm, Sweden. The authors would like to thank the many members of the Future Computing Environments Group at Georgia Tech for their support. Dr. Abowd's work on Classroom 2000 is supported by a National Science Foundation CAREER grant.