10th Annual Conference of the International Speech Communication Association

Brighton, United Kingdom
September 6-10, 2009

Influence of Training on Direct and Indirect Measures for the Evaluation of Multimodal Systems

Julia Seebode (1), Stefan Schaffer (1), Ina Wechsung (2), Florian Metze (3)

(1) Technische Universität Berlin, Germany
(2) Deutsche Telekom Laboratories, Germany
(3) Carnegie Mellon University, USA

Finding suitable evaluation methods is an indispensable task during the development of new user interfaces, as no standardized approach has so far been established, especially for multimodal interfaces. In the current study, we used several data sources (direct and indirect measurements) to evaluate a multimodal version of an information system, tested on trained and untrained users. We investigated the extent to which the different types of data showed concordance concerning the perceived quality of the system, in order to derive clues as to the suitability of the respective evaluation methods. The aim was to examine, if widely used methods not originally developed for multimodal interfaces are appropriate under these conditions, and to derive new evaluation paradigms.

Full Paper

Bibliographic reference.  Seebode, Julia / Schaffer, Stefan / Wechsung, Ina / Metze, Florian (2009): "Influence of training on direct and indirect measures for the evaluation of multimodal systems", In INTERSPEECH-2009, 300-303.