8th Annual Conference of the International Speech Communication Association

Antwerp, Belgium
August 27-31, 2007

A Method for Evaluating Task-Oriented Spoken Dialog Translation Systems Based on Communication Efficiency

Toshiyuki Takezawa (1), Masahide Mizushima (2), Tohru Shimizu (1), Genichiro Kikui (2)

(1) NICT, Japan
(2) NTT Corporation, Japan

We propose a method for measuring communication efficiency from the viewpoint of conveying essential information in task-oriented spoken dialog translation. We present the results of one dialog experiment using speech-to-speech translation systems and a similar experiment using the Wizard of Oz method, which was carried out using hidden interpreters instead of a speech-to-speech translation system. We also present the relative performance score of the speech-to-speech translation system, which was obtained by comparing the machine's performance with that of humans, i.e., hidden interpreters. Finally, we discuss the relationship between users' linguistic behavior and system performance. We found that users of the system tended to make shorter utterances without decreasing the number of essential items needed to achieve a task and to improve transmission efficiency by using strategy to control dialogs.

Full Paper

Bibliographic reference.  Takezawa, Toshiyuki / Mizushima, Masahide / Shimizu, Tohru / Kikui, Genichiro (2007): "A method for evaluating task-oriented spoken dialog translation systems based on communication efficiency", In INTERSPEECH-2007, 1342-1345.