Research Statement

Automated Capture

Much of our life in business and academia is spent listening to and recording, more or less accurately, the events that surround us, and then trying to remember the important pieces of information from those events at some later time. There is clear value, and potential danger, in using computational resources to augment the inefficiency of human record-taking. This is especially true when there are multiple streams of related information in the live experience that are virtually impossible to capture as a whole through manual effort. Computational support for capture can automate the explicit and implicit relationships between related, but separately generated, streams of information. A rich record of a group interaction can support later access to aid in recalling teh meaning or significance of past events.

To better understand the impact of automated capture in our everyday lives, we need tools that seamlessly capture independent streams of information, automate the temporal and semantic relationships between those streams, and provide flexible, accessible and socially appropriate interfaces to the experience. Suites of these tools can remove the burden of doing something humans are not good at (e.g., recording) so that they can focus attention on activities they are good at (e.g., indicating relationships, summarizing, and interpreting).

Our experience in automated capture of live experiences comes mainly from its application in an educational setting through the Classroom 2000 project, a system that has been in use to capture university lectures for the past three years. One of our main early contributions to automated capture research was to develop a framework for capture applications that divided the problem into four stages, pre-production, live capture, post-production and access [C.9]. We have gained an unprecedented level of experience using and evaluating this capture framework in the classroom, and in classrooms at other universities. A history of the project has been summarized in [J.5], extending preliminary evaluation results documented in [C.9] and [C.16] and techniques for the visualization and access to multiple simultaneously recorded streams of information [C.18]. A comprehensive evaluation of the impact of capture in this educational setting, based on distinguished lectures presented over the past year (see [I.1], [I.2], [I.3]) and collected from surveys of over 600 students and 22 instructors, extensive analysis of logs of system use and other evaluation techniques is currently in preparation and will be submitted for publication by the end of August 1999.

One clear outcome of our evaluations is the lack of capabilities in the access phase to support more long-term benefits. To this end, we have extended the capture and access framework to encompass the kinds of activities that occur long after a captured session and blur the line between preparation, live recording, post-production and access for a given live experience. These extensions are detailed in recent submissions discussing dynamic, searchable interfaces to classroom experiences [JS.1], anchoring collaborative discussion spaces within captured lectures [CS.1] and providing scalable visualization and control mechanisms for accessing many hours of captured experience [TR.16].

Most of our experience with capture focussed on capturing the public or shared live experience. Several researchers have looked at capture exclusively for the individual. Nobody had investigated capture in which both public and personal experiences are involved, and this is a rich research problem for both the capture and access phases. This occurs naturally in the classroom domain, as a lecturer presents public information on a topic and students personalize that information through their own notes. Initial experiments in classes attempted to support both the private and public capture, but they failed due to inappropriate private capture technology and inadequate networking (see [C.9] and [J.5] for details). We have developed a system that works within Classroom 2000 to closely integrate student notes with the public lecture experience, and this work is detailed in [C.22] and [C.26].

Beyond the classroom, we have also applied capture to informal encounters [TR.10], architectural design meetings [TR.8] and collaborative workshops. We have also begun to explore workload issues that would help to predict the extent to which Classroom 2000 and other related media-intensive educational applications can be scaled to support an entire campus community. These workload calculations have been calibrated using live data collected from the instrumented use of Classroom 2000 over the past year, as documented in [TR.11] and [CS.2]. An interesting question which we are able to observe is how usage patterns change as students become used to the services of Classroom 2000 over time.

Context-Aware Computing

The richness of human-human interaction is aided greatly by our ability to interpret the context of a given situation and adjust our behavior accordingly. Science fiction visions such as Star Trek suggest the same richness of interaction between humans and the environment, however, today we do not have easy ways of enabling the environment to sense context and wrap that knowledge into adaptable behavior. The area of context-aware computing is aimed at providing easier ways to enable ubiquitous computing applications to sense and react to appropriate cues from the people and environment. Some of the first examples of context-aware behavior sensed the location of users and adapted handheld applications appropriately according to changes in location. We built one of the first demonstrations of a tour guide application, Cyberguide, that provide location-aware cues to a traveller for indoor and outdoor tours. Several conference publications discuss this prototype (see [C.7] and [C.8]) and the entire project is discussed in a journal article [J.2].

Many context-aware computing researchers have examined location-awareness, so we wanted to explore other forms of context. In the Cyberdesk project, we demonstrated how personal productivity tools distributed across a network and on desktop and handheld devices could be automatically integrated based on the context of what single piece of information the user was focussed on. For example, a name in an e-mail message could automatically launch the search for address information on that individual using any of a number of Internet-based directory services and could suggest to the user a one-step action to include that address in a personal contact database contained on a handheld personal digital assistant. One of the novel contributions in this work was an extensible architectural framework for creating inferences from a single piece of information to the variety of related information tidbits that a human would normally associate with it. The initial prototype was presented at several conferences (see [C.11], [C.12] and [C.13]) with a full discussion provided in a journal article [J.3].

Having built significant context-aware computing applications, we understood the potential power of these systems for everyday ubiquitous computing. We also understood why so few context-aware applications were making their way out of research labs and into continual use. Context-aware applications are very difficult to build and the majority of them are done as one-off prototype demonstrations that would not stand the test of time. A major reason for this is that there were very little software engineering benefits applied to the design and implementation of context-aware applications. In the past two years, we have devoted effort to applying software engineering principles to context-aware computing. The main principle is one of separation of concerns. A context-aware application gains its knowledge of the world from sensing technology. Sensors are becoming increasingly powerful and affordable, to the point where we can consider instrumenting many of the spaces we live and work in to sense critical activities. However, applications that would benefit from this sensed information from the physical world currently have to know too many of the details of physical sensing in order to use the information. We have developed a framework for separating the physical aspects of sensing from the application semantics that reacts to changes in sensed context. This Context Toolkit, first described in [C.21] creates a persistent layer, a context layer, between physical sensors and context-aware applications and greatly eases the design and evolution of the applications themselves [TR.13]. Design is eased because the toolkit provides effective abstractions for sensed context, interpretations between context representations and aggregation of context to represent significant entities (such as people or places) from which the application programmer can develop. Evolution of an application is eased because changes to the physical sensing capabilities are shielded from the applications themselves.

As a result of the Context Toolkit, we have been able to easily modify existing applications to make them context aware. For example, we took a capture application similar to Classroom 2000 and quickly adapted it to be more suitable for informal meetings. In the classroom, the context of when a captured session occurred was fixed by a static and well-known class schedule. For informal meetings, it is never known in advance when a meeting will occur. Rather, the context of a meeting must be sensed from activity in a room or around a whiteboard, as demonstrated in the DUMMBO system [TR.10]. This work also demonstrates the close tie between context-aware computing and automated capture. In short, a capture application is one class of context-aware application in which information is stored and tagged with contextual cues (e.g., time, place, people) to facilitate later access. With this link between context-awareness and capture, we built a complex mobile capture application, the Conference Assistant, that provides capture and access features we were never able to do within the framework of Classroom 2000. This work is detailed in a conference paper [C.25] and is the subject of a journal article currently in preparation.

The major demonstration vehicle for the Context Toolkit will be to support ubiquitous sensing in a home environment, through a new an ambitious project called the Aware Home [C.24]. Thousands of sensors will be deployed to provide the information needed by various applications to know where the occupants are located within the house, what they are doing and what the appropriate mechanism is at any place and time to deliver information to those occupants. Specific applications will support care of the elderly within their own home (the Aging in Place project lead by Mynatt and Essa in the FCE Group) and the location of lost objects (Finding Lost Objects, lead by Abowd and Newstetter). In the same way that the focussed application to education in Classroom 2000 provided us great insights into automated capture, we expect the application of sensing to home life to provide a depth of understanding for context-aware computing.

Practical Software Architecture Analysis

Prior to coming to Georgia Tech, I was a post-doctoral research associate at Carnegie Mellon University, where I was engaged in software architecture research. As my own and others' research and development activities confirm, the high-level design, or architecture, of a software system greatly impacts the understandability and long-term evolution of that system. In short, structure matters. One significant contribution to software architecture development was a formalization of architectural style, best summarized in a journal article with Allen and Garlan [J.1].

In collaboration with colleagues at the Software Engineering Insitute, I developed a more practical technique for doing systematic evaluations of software architectures to assess their merits against a changing set of functional and nonfunctional requirements. This scenario-based evaluation technique is called SAAM (for Software Architecture Analysis Method) and has been described in conference [C.2], journal [J.2] and book [B.2] form.

The SAAM technique is a cornerstone to a large DARPA-sponsored research effort among the Software Engineering faculty in the College of Computing. My own contributions to this effort are to better support the natural architectural rationale capture activities of live SAAM sessions through use of automated capture and access technology, as described in [C.20] and to provide better input about the software architecture of a real system through a process we call architectual synthesis and described in two recent conference publications ([C.27] and [C.28]). By systematically extracting perspectives of a software system with the help of existing reverse engineering tools and then comparing the results of those perspectives, we can create through a semi-automated process a more accurate architectural representation that is faithful to the underlying implementation and its designers intentions.

Selected Publications by Abowd

The following are a subset of Dr. Abowd's publications that are significant with respect to the research statement outlined above. A complete listing of publications can be found in Abowd's curriculum vitae.