In order to evaluate the quality of telephone-based interactive systems, two approaches are commonly followed. Firstly, system and user behavior are logged, transcribed and annotated, in order to quantify the performance of the system components and the flow of the interaction between user and system in a parametric way. Secondly, the entire system is evaluated from a users point of view, with the help of questionnaires and quantitative rating scales. For both approaches, recommendations have been issued, defining interaction parameters and the practical set-up of experiments with human test subjects. In addition, prediction algorithms have been proposed to map interaction parameters to subjective user judgments, thus providing quality estimations without relying on user judgments. The present contribution describes what has been reached for each of the approaches, but also the limitations of each methodology. On the basis of experimental data collected with two exemplary systems, shortcomings are identified and future research directions are outlined.
Cite as: Möller, S. (2005) Evaluating telephone-based interactive systems. Proc. Applied Spoken Language Interaction in Distributed Environments (ASIDE 2005), paper 42 (keynote paper)
@inproceedings{moller05_aside,
author={Sebastian Möller},
title={{Evaluating telephone-based interactive systems}},
year=2005,
booktitle={Proc. Applied Spoken Language Interaction in Distributed Environments (ASIDE 2005)},
pages={paper 42 (keynote paper)}
}