Ubiquitous computing, educational application, capture, integration and access of multimedia

One of the potential features of a ubiquitous computing environment is that it could be used to record our everyday experiences and make that record available for later use. Indeed, much of our everyday lives is spent listening to and recording, more or less accurately, the events that surround us, only to have that one important piece of information elude us when we most need it.

There is a value to using computational resources to augment the inefficiency of human record-taking, especially when there are multiple streams of related information generated by a group of people. It is virtually impossible for a human to record all aspects of such a rich interactive experience. Computational support can also automate the relationships between various streams of information, so that one stream of information, such as notes taken during a meeting, can be used to index into another stream, such as the audio track from the same meeting. In this way, an integrated, multimedia record of an experience can aid in the recall of the meaning or significance of past events.

Over the past two years, we have examined this problem within the context of university education, in a project called Classroom 2000 [1, 2]. The objective of the Classroom 2000 project is to create an environment to capture as much of the university classroom experience as possible. We have developed tools to automate the production of on-line lecture notes that integrate the various streams of information common to the classroom experience. Interested readers can find out more on the Classroom 2000 project and experience a number of the tools and captured lectures via our Web site (http://www.cc.gatech.edu/fce/c2000).

Our work has been influenced by the seminal research on capture, integration and access done at Xerox PARC [3, 4]. Some of the novel aspects of our work are attention to universal access and support for a large number of users in everyday classroom situations.

In this extended abstract, we will describe tools for capturing lecture information and accessing on-line lectures via the Web.

We want to capture as much of the activity that defines the classroom experience. We view a lecture as a collaborative multimedia authoring session, the purpose of which is to generate multimedia content that assists a student in reliving a lecture. During a lecture, we use an electronic whiteboard (the hardware we use is a Liveboard) running a Java-based client-server system, ZenPad, to present slides and allow for annotation by the teacher. A typical session in ZenPad is shown in Figure 1. The teacher simply annoates static GIF images with a pen.

The interface to ZenPad is purposely minimal. The two important features of our software that are not available in commercial alternatives are:

• production of Web-accessible artifacts, namely GIF images; and

• a timestamped log of important interaction, such as penstrokes with the electronic whiteboard.

This last feature enables subsequent integration with recorded digital audio or video. We use commercial streaming audio and video technology (RealAudio and Vxtreme) to produce a live digital encoding of what is seen and heard in the classroom.

In the access phase, students and teachers are provided a Web interface to browse past lectures. The ZenPad system shown in Figure 1 provides a browse-only mode that is suitable for this purpose, but we have also produced a number of HTML interfaces that are more universally accessible to our intended student population. In all cases, there is the ability to jump from the displayed lecture material to the audio for that lecture. Different prototypes provided different indexing capabilities. For example, one interface provides thumbnails of all lecture slides to facilitate browsing. A particular slide could be magnified, thus allowing a student to jump to the audio stream of the actual lecture at the time the slide was entered. In browse mode, ZenPad allows indexing down to every penstroke. An HTML version of the slides exported from ZenPad provides client side image maps that allow for word-level integration of handwriting and audio.

We have also experimented with the notion of an interactive timeline associated with a class. Figure 2 shows a sample interface. On the left is a timeline to mark from beginning to end of the lecture. This timeline is decorated with two kinds of events, creation of a slide in ZenPad and visits to URLs in an accompanying Web browser in the classroom. Clicking on either of these event decorations pulls up an image of the slide or URL. The ZenPad slides have image maps that control the video window in the bottom corner of the screen. The timeline also controls the video playback.

Near-term efforts on ZenPad will create a larger-scale electronic whiteboard the size of a traditional classroom whiteboard. We are also providing alternative interfaces for students to capture notes during the class using a variety of tablet computers and handheld PDAs. We are continuing to experiment with alternative interfaces to access captured lecture materials. We are particularly interested in how speech and vision technology can assist in the content-based browsing and retrieval of material within and across lectures.