1 INTRODUCTION

1.1 Motivating the Problem

Computer interfaces which support more natural human forms of communication (e.g. handwriting, speech, and gestures) are beginning to supplement or replace elements of the GUI paradigm. These interfaces are lauded for their low learning curves and their ability to support tasks such as authoring and drawing without drastically changing their structure. Additionally, they can be used by people with disabilities that make the traditional mouse and keyboard less accessible.

Unfortunately, these new interfaces come with a new set of problems --they make mistakes. When errors occur, the initial reaction of system designers is to try to eliminate them, for example by improving recognition accuracy. This is often a difficult task --Buskirk & LaLomia (1995) found that an improvement of 5-10% is necessary before the majority of people will even notice a difference in a speech recognition system.

Worse yet, eliminating errors may not be possible. Even humans make mistakes when dealing with these same forms of communication. As an example, consider handwriting recognition. Even the most expert handwriting recognizers (humans) can have a recognition accuracy as low as 54% when looking at word fragments without the benefit of their context (Schomaker, 1994). Human accuracy increases to 88% for cursive handwriting (Schomaker, 1994), and 96.8% for printed handwriting (Frankish et al., 1995), but it is never perfect. This evidence all points to the conclusion that computer handwriting recognition will never be perfect.

Computer-based recognizers are even more error prone than humans. The data they start with is often less fine-grained than that which humans are able to sense. They have less processing power. And variables such as vocal fatigue can cause usage data to differ significantly from training data, causing reduced recognition accuracy over time in speech recognition systems (Frankish et al., 1992).

On the other hand, recognition accuracy is not the only determinant for user satisfaction. Both the complexity of error recovery dialogues (Zajicek & Hewitt, 1990), and the amount gained for the effort (Frankish, et al., 1992), affect user satisfaction. For example, Frankish found that users were less frustrated by recognition errors when the task was to enter a command in a form than when they were writing journal entries. He suggests that this is because the pay-back for entering a single word in the case of a command is much larger than in a paragraph of a journal entry when compared with the effort of entering the word.

Error handling is not a new problem. In fact, it is endemic to the design of computer systems which attempt to mimic human abilities. Research in the area of error handling for recognition technologies must assume that errors will occur, and then answer questions about the best ways to deal with them. The goal of this paper is to present a survey of existing research in discovering and correcting errors in recognition based interfaces.

1.2 Defining The Area

Our survey has have identified five key research areas for error handling of recognition-based interfaces.

Error reduction

Error reduction involves research into improving recognition technology in order to eliminate or reduce errors. It has been the focus of extensive research, and could easily be the subject of a whole paper on its own. Evidence suggests that its holy grail, the elimination of errors, is probably not achievable. And big improvements (5-10%) are required before users even notice a difference (Buskirk & LaLomia, 1995). Because of these facts, we have chosen not to address error reduction in this paper.

Error discovery

Before either the system or the user can take any action related to a given error, one of them has to know that the error has occured. The system may be told of an error through user input, and can help the user to find errors through its output. In addition, system designers have used three techniques to automate error discovery --thresholding, rules, and historical statistics.

Error correction techniques

Just as the user interface is the only way one party can inform the other that an error has occured, it is also the only way that the user can correct an error. We found that current error handling techniques fall into three main categories --choosing a default, encouraging less ambiguous input, and mimicking natural human correction strategies.

Validation of techniques

Validation goes hand in hand with research into error correction techniques. Validation is the only way to determine the effectiveness of different designs. Our survey uncovered research into theoretical issues such as how to compare techniques, and practical results such as which techniques are effective.

Toolkit level support

Toolkits provide reusable components and are most useful when a class of common, similar problems exists. Interfaces for error handling would benefit tremendously from a toolkit which could be used and re-used every time an error prone situation arose. In addition to interface widgets, a toolkit would need to support complete reversibility, and keep track of multiple potential interpretations at once.

  
Figure 1: PenPad's user interface. The words: Penpad; around; both the; all; potential; were all recognized correctly. The darker the word, the surer the recognizer is of this. The word ``interpretations'' was recognized incorrectly. When the user moves the mouse over this word, five alternatives are displayed, shown in the blow-up. The words ``ink, and'' were originally incorrect, but the user was able to select them from a similar set of five potential choices.

In addition to surveying existing work, we are building a platform to test strategies for dealing with segmentation errors, handwriting recognition errors, and gesture recognition errors (see Figure 1). Our system, called PenPad, supports handwriting recognition in the context of personal note-taking. Our motivation for this application is to support note taking and document creation in situations when typing is not an option. This includes mobile settings, and users with repetitive stress injuries or other disabilities which make keyboard typing difficult.

The rest of this paper describes the results of our survey. We discuss research in each of the last four sub-areas mentioned above --error discovery, error correction techniques, validation of techniques, and toolkit level support.