11th Annual Conference of the International Speech Communication Association

Makuhari, Chiba, Japan
September 26-30. 2010

Assessment of Spoken and Multimodal Applications: Lessons Learned from Laboratory and Field Studies

Markku Turunen, Jaakko Hakulinen, Tomi Heimonen

University of Tampere, Finland

In this paper, we present the key lessons learnt from numerous evaluations conducted to measure the quality of spoken and multimodal applications. The issues we address include the relation of laboratory and field studies, long-term and pilot evaluations, unimodality and multimodality, objective and subjective metrics, and user expectations and experiences. We present concrete case studies to discuss the above issues. For example, there are major differences in evaluating speech-only and multimodal systems. Similarly, there are major differences between laboratory and field studies, which need to be considered in successful evaluations.

Full Paper

Bibliographic reference.  Turunen, Markku / Hakulinen, Jaakko / Heimonen, Tomi (2010): "Assessment of spoken and multimodal applications: lessons learned from laboratory and field studies", In INTERSPEECH-2010, 1333-1336.