5th International Conference on Spoken Language Processing
This paper reports the experiences we had in evaluating the ACCeSS system using the EAGLES evaluation metrics both at the input/output (black box evaluation) and component levels (glass box evaluation). Our primary motivation for using the EAGLES metrics was to apply a set of metrics that allow objective measurement of system performance. These measures are used both for diagnostic evaluation of the current version and for progress evaluation of successive system versions. The EAGLES metrics provided very valuable guidelines for our evaluation, but not all the metrics were unproblematic. Especially for diagnostics during development, the addition of subtask success rate proved very helpful.
Bibliographic reference. Hanrieder, Gerhard / Heisterkamp, Paul / Brey, Thomas (1998): "Fly with the EAGLES: evaluation of the "ACCeSS" spoken language dialogue system", In ICSLP-1998, paper 0274.