Third European Conference on Speech Communication and Technology

Berlin, Germany
September 22-25, 1993


The Cost of Errors in a Spoken Language System

Lynette Hirschman, Christine Pao

Spoken Language Systems Group, Laboratory for Computer Science, Massachusetts Institute of Technology, Cambridge, Massachusetts, USA

Spoken language interfaces are rapidly improving in both speed and accuracy. However, miscommunication between system and user still occurs, with the potential for disrupting user/system interchange. This paper examines what happens when such miscommunication occurs: how quickly the user attempts to correct the system's non-understanding, what effect this has on subsequent communication, and how "expensive" miscommunication is in terms of solving a problem. We present a framework for analyzing detection and correction of system errors. Based on data from several experiments, we determined that in over 85% of the cases, the user detected a system misunderstanding in the next turn. For two systems with different overall performance (21% vs. 35% sentence understanding error) but identical response generation capabilities, errors took an average of 1.25 and 1.33 turns to detect, respectively; for a system with modified interaction and feedback strategies (and understanding error of 25%) the average dropped to 1.19 turns. This reduction in error detection time indicates that this measure is sensitive to aspects of the system/user interface beyond simple sentence understanding error.

Full Paper

Bibliographic reference.  Hirschman, Lynette / Pao, Christine (1993): "The cost of errors in a spoken language system", In EUROSPEECH'93, 1419-1422.