5th International Conference on Spoken Language Processing

Sydney, Australia
November 30 - December 4, 1998

Fly with the EAGLES: Evaluation of the "ACCeSS" Spoken Language Dialogue System

Gerhard Hanrieder (1), Paul Heisterkamp (2), Thomas Brey (3)

(1) Daimler-Benz Aerospace AG, Germany
(2) Daimler-Benz AG, Germany
(3) University of Regensburg, Germany

This paper reports the experiences we had in evaluating the ACCeSS system using the EAGLES evaluation metrics both at the input/output (black box evaluation) and component levels (glass box evaluation). Our primary motivation for using the EAGLES metrics was to apply a set of metrics that allow objective measurement of system performance. These measures are used both for diagnostic evaluation of the current version and for progress evaluation of successive system versions. The EAGLES metrics provided very valuable guidelines for our evaluation, but not all the metrics were unproblematic. Especially for diagnostics during development, the addition of subtask success rate proved very helpful.

Full Paper

Bibliographic reference.  Hanrieder, Gerhard / Heisterkamp, Paul / Brey, Thomas (1998): "Fly with the EAGLES: evaluation of the "ACCeSS" spoken language dialogue system", In ICSLP-1998, paper 0274.