10th Annual Conference of the International Speech Communication Association

Brighton, United Kingdom
September 6-10, 2009

Predicting the Quality of Multimodal Systems Based on Judgments of Single Modalities

Ina Wechsung (1), Klaus-Peter Engelbrecht (1), Anja B. Naumann (1), Stefan Schaffer (2), Julia Seebode (2), Florian Metze (3), Sebastian Möller (1)

(1) Deutsche Telekom Laboratories, Germany
(2) Technische Universität Berlin, Germany
(3) Carnegie Mellon University, USA

This paper investigates the relationship between user ratings of multimodal systems and user ratings of its single modalities. Based on previous research showing precise predictions of ratings of multimodal systems based on ratings of single modality, it was hypothesized that the accuracy might have been caused by the participantsí efforts to rate consistently. We address this issue with two new studies. In the first study, the multimodal system was presented before the single modality versions were known by the users. In the second study, the type of system was changed, and age effects were investigated. We apply linear regression and show that models get worse when the order is changed. In addition, models for younger users perform better than those for older users. We conclude that ratings can be impacted by the effort of users to judge consistently, as well as their ability to do so.

Full Paper

Bibliographic reference.  Wechsung, Ina / Engelbrecht, Klaus-Peter / Naumann, Anja B. / Schaffer, Stefan / Seebode, Julia / Metze, Florian / Möller, Sebastian (2009): "Predicting the quality of multimodal systems based on judgments of single modalities", In INTERSPEECH-2009, 1827-1830.