15th Annual Conference of the International Speech Communication Association

September 14-18, 2014

Evaluating Coherence in Open Domain Conversational Systems

Ryuichiro Higashinaka, Toyomi Meguro, Kenji Imamura, Hiroaki Sugiyama, Toshiro Makino, Yoshihiro Matsuo

NTT Corporation, Japan

We propose a method for evaluating coherence between user utterances and those generated from open domain conversational systems. Our aim is to make it possible for such systems to ascertain whether utterances generated from them are appropriate to the context before generation so that possible breakdown in conversation arising from inappropriate utterances can be avoided. In our method, we train a classifier that distinguishes a pair of a user utterance and that generated from a system as coherent or incoherent by using various pieces of information related to dialogue exchange, such as dialogue acts, question types, and predicate-argument structures. Experimental results show that our method significantly outperforms the baseline, confirming its effectiveness.

Full Paper

Bibliographic reference.  Higashinaka, Ryuichiro / Meguro, Toyomi / Imamura, Kenji / Sugiyama, Hiroaki / Makino, Toshiro / Matsuo, Yoshihiro (2014): "Evaluating coherence in open domain conversational systems", In INTERSPEECH-2014, 130-134.