 |
Applied Spoken Language Interaction in Distributed Environments (ASIDE 2005)ITRW and
COST278 Final Workshop |
 |
|
Applied Spoken Language Interaction in Distributed Environments (ASIDE 2005)ITRW and
COST278 Final Workshop |
|
In order to evaluate the quality of telephone-based interactive systems, two approaches are commonly followed. Firstly, system and user behavior are logged, transcribed and annotated, in order to quantify the performance of the system components and the flow of the interaction between user and system in a parametric way. Secondly, the entire system is evaluated from a user’s point of view, with the help of questionnaires and quantitative rating scales. For both approaches, recommendations have been issued, defining interaction parameters and the practical set-up of experiments with human test subjects. In addition, prediction algorithms have been proposed to map interaction parameters to subjective user judgments, thus providing quality estimations without relying on user judgments. The present contribution describes what has been reached for each of the approaches, but also the limitations of each methodology. On the basis of experimental data collected with two exemplary systems, shortcomings are identified and future research directions are outlined.
Bibliographic reference. Möller, Sebastian (2005): "Evaluating telephone-based interactive systems", In ASIDE-2005, paper 42.