EUROSPEECH 2003 - INTERSPEECH 2003
This paper describes a method for creating an evaluation measure for discourse understanding in spoken dialogue systems. Discourse understanding means utterance understanding taking the context into account. Since the measure needs to be determined based on its correlation with the system's performance, conventional measures, such as the concept error rate, cannot be easily applied. Using the multiple linear regression analysis, we have previously shown that the weighted sum of various metrics concerning dialogue states can be used for the evaluation of discourse understanding in a single domain. This paper reports the progress of our work: verification of our approach by additional experiments in another domain. The support vector regression method performs better than the multiple linear regression method in creating the measure, indicating non-linearity in mapping the metrics to the system's performance. The results give strong support for our approach and hint at its suitability as a universal evaluation measure for discourse understanding.
Bibliographic reference. Higashinaka, Ryuichiro / Miyazaki, Noboru / Nakano, Mikio / Aikawa, Kiyoaki (2003): "Evaluating discourse understanding in spoken dialogue systems", In EUROSPEECH-2003, 1941-1944.