Should We Leverage Natural-Language Knowledge? An Analysis of User Errors in a Natural-Language-Style Programming Language

Should programming languages use natural-language-like syntax? Under what circumstances? What sorts of errors do novice programmers make? Does using a natural language-like programming language lead to user errors? In this study, we read the entire online interactions of sixteen children who issued a total of 35,047 commands on MOOSE Crossing, an educational MUD for children, We counted and categorized the errors made. A total of 2,970 errors were observed. We define "natural-language errors" as those errors in which the user failed to distinguish between English and code, issuing an incorrect command that was more English-like than the correct one. A total of 314 natural-language errors were observed. In most of those errors, the child was able to correct the problem either easily (41.1% of the time) or with some effort (20.7%). Natural-language errors were divided into five categories. In order from most to least frequent, they are: syntax errors, guessing a command name by supplying an arbitrary English word, literal interpretation of metaphor, assuming the system is keeping more state information than is actually the case, and errors of operator precedence and combination. We believe that these error rates are within acceptable limits, and conclude that leveraging users' natural-language knowledge is for many applications an effective strategy for designing end-user-programming languages.

Authors

Amy Bruckman , Elizabeth Edwards

Citation

Bruckman, Amy and Elizabeth Edwards (1999). "Should We Leverage Natural-Language Knowledge?" Proceedings of CHI 99. New York: ACM Press, pp. 207-214. (25% acceptance rate.)

Download

[ PDF ]