ISCA Archive ICSLP 2000
ISCA Archive ICSLP 2000

Data collection and performance evaluation of spoken dialogue systems: the MIT experience

James Glass, Joseph Polifroni, Stephanie Seneff, Victor Zue

In this paper we report our efforts in data collection and performance evaluation in support of spoken dialogue system development. We describe two understanding metrics called query density and concept efficiency which can be interpreted on a perutterance basis, but which are measured over the course of a dialogue. We also describe the evaluation infrastructure we have developed to support off-line data processing using our GALAXY client-server architecture [8]. We show how we have used these metrics and mechanisms as part of the development of a spoken dialogue system for air-travel information.

S. Seneff, E. Hurley, R. Lau, C. Pao, P. Schmid, and V. Zue, "GALAXY-II: A reference architecture for conversational system development," Proc. ICSLP, 931-934, Sydney, 1998.


Cite as: Glass, J., Polifroni, J., Seneff, S., Zue, V. (2000) Data collection and performance evaluation of spoken dialogue systems: the MIT experience. Proc. 6th International Conference on Spoken Language Processing (ICSLP 2000), vol. 4, 1-4

@inproceedings{glass00_icslp,
  author={James Glass and Joseph Polifroni and Stephanie Seneff and Victor Zue},
  title={{Data collection and performance evaluation of spoken dialogue systems: the MIT experience}},
  year=2000,
  booktitle={Proc. 6th International Conference on Spoken Language Processing (ICSLP 2000)},
  pages={vol. 4, 1-4}
}