Domisilica: Providing Ubiquitous Access to the Home

ABSTRACT

Keywords

INTRODUCTION

We have chosen the kitchen as the place to begin our exploration of computing in the home. The refrigerator, for example, has many uses beyond keeping food cool in the common household: making shopping lists, leaving notes, and posting information and pictures. Domisilica's computational capabilities and network connection allow us to expand these tasks. By attaching a computer display to the front of the refrigerator, we can display virtual notes and web pages which have been "posted" onto the refrigerator. A remote user can view the contents of the refrigerator in a GUI or hear them over the phone. That user can create notes and post them to the refrigerator, or run an application which makes use of the kitchen inventory system to generate a shopping list or produce a potential menu for dinner. See Figure 1 for a idea of the kind of remote interaction we currently support in Domisilica. (See the included video, and the Domisilica web pages for a demonstration of this scenario.

Currently, computing in the home takes two forms: It is centralized in one box where occupants go to run various applications, or it is distributed in small, autonomous chunks in appliances such as microwaves, VCRs, and security/climate control systems. We envision a future in which devices currently common in households are augmented with additional information and functionality, and networked together to provide information and services to remote users.

This scenario presents several interesting problems:

1. Once a real world object or device has been augmented with new services and information, how do you make the user aware of these services, and how do you make these services available to a user through the object itself ("on location")? We believe that this is best done by extending existing affordances of the object when possible, and adding new ones when necessary. As an example, consider a front door which has been augmented with an outdoor camera. One way of extending the affordances of the door might be to replace the peephole with a "hole" that can be rotated and slid around the door in order to control the view of the camera.
2. Universal accessibility is very important both locally and remotely. Homes are lived in by people of all sorts and sizes, including people who are handicapped, older, very young, and people who may have very little computer experience, etc. Our system also needs to be accessible remotely to people without computers and people with very low bandwidth connections and limited services, as well as computing professionals with high-bandwidth personal machines. Here again, the accessibility of the system to different people is an important consideration.

Overview of Paper

RELATED WORK

While people have studied computing in the home ([15],[8]), the focus has generally been on technologies imported from the office place (such as email, the Web, and computer-centric tasks like word-processing). One exception to this is the Neural Network House [10], a home which manages climate control and learns about its user's habits over time. While this research explores a climate control service in depth, it does not address the interface issues we are contending with.

Both the ubiquitous computing community and many entrepreneuring technophiles have explored ways of adding new computing power throughout our physical environments. (See the July 1993 CACM (titled "Back to the Real World").) Augmented reality ([17], [4],[16], [18]), represents one approach to giving users access to new services and information. The DigitalDesk [18] project is a good example of how the affordances of paper can be extended to provide new services. The ParcTab/Pad [17] ubiquitous computing project involves adding new computing objects to the environment rather than extending already existing objects into the computing world. Fitzmaurice [4] describes a portable, personal palmtop display which can be used to view augmented aspects of real world objects, and Feiner et. al [3] use a wearable display to show virtual information overlayed on real objects. While these are a valid approaches, they require the user to carry the interface. As an alternative, we have chosen to build the input and output into the augmented device. This type of interface is exemplified by the work of Ishii et. al in graspable user interfaces and tangible bits ([6] [5].)

Much of the research done in supporting disabled users ([7],[14],[2],[13]) is very applicable to Domisilica. Work involving blind users can support both our goal of universal access, and can also be applied to the development of non-visual interfaces such as phone-based access. The Dual[14] project provides a way of generating both a visual and non-visual interface to the same application in an integrated fashion, while Mercator [2] automatically generates an audio interface from an X-based GUI. Emacspeak [13], similar to Dual, can present both visual (text-based) and non-visual interfaces simultaneously. Domisilica follows the philosophy of Dual: that non GUI interfaces are best developed independently with metaphors appropriate to the medium. We feel, however, that there are many different modes of UI's which should be supported, each with different metaphors, not just two. These include GUIs; visual but non-graphical UIs (eg MOOs (a MOO is object-oriented, extensible database built to support "shared computing with a powerful real world metaphor." [1]), emacs); audio and phone-based UIs (eg conversational metaphors), and many others.

THE DOMISILICA SYSTEM

The Java toolkit provides support for building interactors for objects in the database (which generally correspond to devices, places, or services in the home). It also supports a communication protocol for requesting information and receiving events from the MOO. The GUI aspects of the package are built on top of SubArctic. (SubArctic, like Mercator [2] can generate audio interfaces, but the preferred method for non-visual access is via the conversational, text-only interface automatically provided by the MOO). The GUI includes a window (the "Application Manager") which will dynamically load all of the interactors associated with objects in the MOO model which are visible to the user. This means that users can add new functionality to the database simply by writing a java class (an interactor, in the GUI case) which implements that functionality, and setting a special property of the corresponding database object to contain the name of the new class. The standard GUI system will then dynamically load that new interactor, incorporating it into the user's display on the fly. Our GUI application manager can be run from the WWW as an applet or locally as a Java application.

As an example of the real world side of things, we have implemented a kitchen inventory system which uses a bar code reader and an image recognizer to identify food items as they are unpacked. Note that while this may support additional functionalities, it is not in and of itself a way to access new data or call new functions. It is simply a way of informing the computer of changes in state. Ideally this should be done auto-magically. As the system stands, it still requires some overhead from the user, who must run each food item through the scanner or recognizer before putting it away. We feel that this is not unreasonable since most people start at a central place (a pile of bags) when unloading groceries, provided we can show them some gain for the extra work. Ultimately we hope to embed scanning technology in the places where food is stored. Supermarkets are already working on technology which can scan a whole shopping cart at once. Why not a whole pantry or refrigerator?

When designing this system, we chose a centralized model of the real world because it simplifies the task of providing a representation of the current state of services available to the user. There are arguments for additional robustness in distributed systems, but Domisilica can also be integrated with distributed computing in the house if each individual appliance is responsible for updating their status in the database with log data at regular intervals.

PROVIDING ACCESS TO COMPUTATIONALLY ENHANCED REAL WORLD OBJECTS

On Location Access

Remote Access

GUI for full-bandwidth connections

text for limited-bandwidth connections or small displays

audio output for low-vision users, to provide "atmosphere", and eventually for any situation where audio might be more appropriate.

audio-to-phone output for receiving status updates via a portable system (we are working on input, at which point users will be able to phone the system and interact with it using a combination of DTMF and voice).

ONE SYSTEM TO SUPPORT MULTIPLE MODALITIES, METAPHORS, AND USERS

This approach is facilitated by the fact that all of our functionality is stored in a database (MOO objects are programmable, so in addition to storing data, they store functionality). Interfaces are just a view of the properties of objects (data) plus interactors for accessing their functionality. One very important aspect of this work is that different interfaces present the user with a metaphor tailored for the modality of interaction ([14], [20], [7]). We have a slight advantage here over other applications; there is an obvious semantic metaphor that holds the system together: the mapping to the real world (rooms in a house containing various appliances). This metaphor is used by our version of a Window Manager or "Application Manager". It gives the user a way to access any of the many applications available in a given room and to navigate to different rooms. Even so, this metaphor must be tailored to the situation, and different representations of a room or set of rooms are appropriate in different settings. For example, one interface may draw a three-dimensional picture of a room, while another might simply map real world places to folders and appliances to recognizable icons, and a third might present a conversational interface involving textual descriptions. Also, the real world mapping metaphor does not help as much at the application level. Each application may be built by a different person, and must support multiple modalities and metaphors. This is especially true for functions which have been added to the real world -- for example, a function which visualizes the changes in population of a room over time might take a very different form in a GUI than "on location" or in an audio interface. Our solution to this problem is to provide toolkits for different modalities. Toolkits for GUI, audio, and text-based access are described in the following sections.

Toolkit for building GUI interfaces

Programmers can subclass from these interactors to provide special functionality, or use as-is if they simply wish to represent state information about the domicile. Each MOO object is represented on-screen either as a dynamically-loaded Java class, or a pointer to a URL. Most MOO objects are shown as icons which can be used to identify the real world object corresponding to that MOO interface object. The icon, Java classname, and other UI information, along with functions implementing any special functionality are all stored in the MOO. The Java toolkit simply parses that data (we provide a parser class and a connection to the MOO along with a protocol for requesting and receiving information) and makes the appropriate changes in the UI.

Text-based access

Audio access

FUTURE WORK AND CONCLUSIONS

Persistence in UI's: The Domisilica system is a persistent database which may have users connected at any time, or include a persistent UI. When a new appliance is added, or an interface to an existing appliance is updated, currently connected users should not have to exit and reconnect, and persistent interfaces should automatically update to reflect the new state of the database.

Security: If Domisilica is to come into use in a real house, we need to make some guarantees about security, and our user interfaces need to support those guarantees. Currently, users are required to have a password to access the system, but that's the extent of our security. In the future, we need to make it possible for members of the same household to have secure spaces which other family members cannot enter, and to provide more stringent authentication in order to make the system more secure from outside hackers.

Concurrency and Events: Although all of the current users support multiple users, our current concurrency model is very naive. We plan to build a more robust system modeled on that of Jupiter [11]. We also plan to build a more sophisticated event handler by integrating real world events into the SubArctic user-interface event model.

Real World affordances: Time and thought need to be spent on generalizing the ways in which the affordances of real objects might be extended and providing a toolkit to build these interfaces. The toolkit's development is complicated by the need for physical interaction devices, and a complete toolkit should probably come with sensors and "physical interactors" as well as virtual or "computational" interactors.

A real home: In order to really test the effectiveness of the Domisilica system, we need to integrate it with a real home. We currently have one volunteer who has already implemented a distributed system in his home. We plan to begin integrating with his home this spring.

ACKNOWLEDGMENTS

REFERENCES