Ninth International Conference on Spoken Language Processing

Pittsburgh, PA, USA
September 17-21, 2006

MeMo: Towards Automatic Usability Evaluation of Spoken Dialogue Services by User Error Simulations

Sebastian Möller (1), Roman Englert (1), Klaus Engelbrecht (1), Verena Hafner (1), Anthony Jameson (2), Antti Oulasvirta (1), Alexander Raake (1), Norbert Reithinger (2)

(1) Technische Universität Berlin, Germany; (2) DFKI GmbH, Germany

Proper usability evaluations of spoken dialogue systems are costly and cumbersome to carry out. In this paper, we present a new approach for facilitating usability evaluations which is based on user error simulations. The idea is to replace real users with simulations derived from empirical observations of users’ erroneous behavior. The simulated errors must cover both system-driven errors (e.g., due to poor speech recognition) as well as conceptual errors and slips of the user, because neither alone is predictive of perceived usability. The simulation is integrated into a workbench which produces reports of typical and rare errors, and which allows usability ratings to be predicted. If successful, this workbench will help designers in making choices between system versions and lower testing costs at early phases of development. Challenges to the approach are discussed and solutions proposed.

Full Paper

Bibliographic reference.  Möller, Sebastian / Englert, Roman / Engelbrecht, Klaus / Hafner, Verena / Jameson, Anthony / Oulasvirta, Antti / Raake, Alexander / Reithinger, Norbert (2006): "Memo: towards automatic usability evaluation of spoken dialogue services by user error simulations", In INTERSPEECH-2006, paper 1131-Wed2WeS.6.