7th International Conference on Spoken Language Processing

September 16-20, 2002
Denver, Colorado, USA

DARPA Communicator: Cross-System Results for the 2001 Evaluation

Marilyn A. Walker (1), Alexander I. Rudnicky (2), Rashmi Prasad (3), John Aberdeen (4), Elizabeth Owen Bratt (5), John S. Garofolo (6), Helen Hastie (3), Audrey N. Le (6), Bryan Pellom (7), Alex Potamianos (8), Rebecca Passonneau (3), Salim Roukos (9), Gregory A. Sanders (6), Stephanie Seneff (10), David Stallard (11)

(1) AT&T Labs - Research, USA; (2) Carnegie Mellon University, USA; (3) AT&T Labs, USA; (4) MITRE, USA; (5) SRI, USA; (6) National Institute of Standards and Technology, USA; (7) University of Colorado at Boulder, USA; (8) Lucent Technologies, USA; (9) IBM, USA; (10) MIT Laboratory for Computer Science, USA; (11) BBN Technologies, USA

This paper describes the evaluation methodology and results of the 2001 DARPA Communicator evaluation. The experiment spanned 6 months of 2001 and involved eight DARPA Communicator systems in the travel planning domain. It resulted in a corpus of 1242 dialogs which include many more dialogues for complex tasks than the 2000 evaluation. We describe the experimental design, the approach to data collection, and the results. We compare the results by the type of travel plan and by system. The results demonstrate some large differences across sites and show that the complex trips are clearly more difficult.

Full Paper

Bibliographic reference.  Walker, Marilyn A. / Rudnicky, Alexander I. / Prasad, Rashmi / Aberdeen, John / Bratt, Elizabeth Owen / Garofolo, John S. / Hastie, Helen / Le, Audrey N. / Pellom, Bryan / Potamianos, Alex / Passonneau, Rebecca / Roukos, Salim / Sanders, Gregory A. / Seneff, Stephanie / Stallard, David (2002): "DARPA communicator: cross-system results for the 2001 evaluation", In ICSLP-2002, 269-272.