Sixth International Conference on Spoken Language Processing
(ICSLP 2000)

Beijing, China
October 16-20, 2000

Data Collection and Performance Evaluation of Spoken Dialogue Systems: The MIT Experience

James Glass, Joseph Polifroni, Stephanie Seneff, Victor Zue

Spoken Language Systems Group, Laboratory for Computer Science, Massachusetts Institute of Technology, Cambridge, MA, USA

In this paper we report our efforts in data collection and performance evaluation in support of spoken dialogue system development. We describe two understanding metrics called query density and concept efficiency which can be interpreted on a perutterance basis, but which are measured over the course of a dialogue. We also describe the evaluation infrastructure we have developed to support off-line data processing using our GALAXY client-server architecture [8]. We show how we have used these metrics and mechanisms as part of the development of a spoken dialogue system for air-travel information.


  1. S. Seneff, E. Hurley, R. Lau, C. Pao, P. Schmid, and V. Zue, "GALAXY-II: A reference architecture for conversational system development," Proc. ICSLP, 931-934, Sydney, 1998.

Full Paper

Bibliographic reference.  Glass, James / Polifroni, Joseph / Seneff, Stephanie / Zue, Victor (2000): "Data collection and performance evaluation of spoken dialogue systems: the MIT experience", In ICSLP-2000, vol.4, 1-4.