7th International Conference on Spoken Language Processing

September 16-20, 2002
Denver, Colorado, USA

A Method for Evaluating Incremental Utterance Understanding in Spoken Dialogue Systems

Ryuichiro Higashinaka, Noboru Miyazaki, Mikio Nakano, Kiyoaki Aikawa

NTT Corporation, Japan

In single utterance understanding, which does not include discourse understanding, the concept error rate (CER), or the keyword error rate, has been widely used as an evaluation measure for utterance understanding. However, the CER cannot be used for evaluating systems that understand user utterances based on previous user utterances. In this paper, we propose a method for evaluating incremental utterance understanding, which involves speech recognition, language understanding and discourse processing in spoken dialogue systems, by finding a measure that correlates closely with the systemís performance based on dialogue states and their way of update. We defined dialogue performance by task completion time, and performed a multiple linear regression analysis using task completion time as the explained variable and various metrics concerning dialogue states as explaining variables. The obtained multiple regression model fits comparatively well and shows validity as an evaluation measure.


Full Paper

Bibliographic reference.  Higashinaka, Ryuichiro / Miyazaki, Noboru / Nakano, Mikio / Aikawa, Kiyoaki (2002): "A method for evaluating incremental utterance understanding in spoken dialogue systems", In ICSLP-2002, 829-832.