8th European Conference on Speech Communication and Technology

Geneva, Switzerland
September 1-4, 2003


Towards an Evaluation Standard for Speech Control Concepts in Real-World Scenarios

Jens Maase (1), Diane Hirschfeld (2), Uwe Koloska (2), Timo Westfeld (3), Jorg Helbig (3)

(1) Bosch und Siemens Hausgerate GmbH, Germany
(2) voice INTER connect GmbH, Germany
(3) MediaInterface Dresden GmbH, Germany

Speech control is still mainly evaluated through statistical performance measures (recognition rate, insertion rate, etc.) considering the performance of a speech recognizer under laboratory or artificial noise conditions. All these measures give no idea about the practical usability of a speech interface, since practical aspects concern more than operational aspects of the speech recognizer inside a product. Since it was felt, that no evaluation standard so far fulfills the practical requirements for speech controlled products, this paper aims in the establishment of an open design- and evaluation standard for speech control concepts in real-world scenarios. First, the behaviour of the users and the normal environmental conditions (typical noises) were evaluated in usability experiments. Data recordings were conducted, trying to capture these typical usage requirements in a special corpus (Apollo-corpus). Finally, a set of standard desktop-, as well as embedded speech recognizers were tested for their performance under these real world conditions.

Full Paper

Bibliographic reference.  Maase, Jens / Hirschfeld, Diane / Koloska, Uwe / Westfeld, Timo / Helbig, Jorg (2003): "Towards an evaluation standard for speech control concepts in real-world scenarios", In EUROSPEECH-2003, 1553-1556.