First ISCA ITRW on Auditory Quality of Systems

April 23-25, 2003
Akademie Mont-Cenis, Germany

Assessing the Usability of a Dialogue Management System Designed in the Framework of a rapid Dialogue Prototyping Methodology

Martin Rajman, Andréa Rajman, Florian Seydoux, Alex Trutnev

Artificial Intelligence Laboratory, Swiss Federal Institute of Technology, Lausanne, Switzerland

The goal of this contribution is to present how the notion of dialogue management evaluation has been integrated in the rapid dialogue prototyping methodology we have designed and experimented in the framework of the InfoVox project.

We first describe the proposed rapid dialogue prototyping methodology, decomposed into 5 consecutive main steps: (1) producing the task model; (2) deriving the initial dialogue model; (3) using a Wizard-of-Oz experiment to instantiate the initial dialogue model; (4) using an internal field test to refine the dialogue model; and (5) using an external field test to evaluate the final dialogue model.

In a second part, we more specifically focus on evaluation. We first describe the integration of various evaluation aspects in the design steps corresponding to the internal and external field tests. Then, we more specifically concentrate on system usability assessment: we first describe the retrospective trend and correlation analyses that have been integrated in the field tests and then show how the obtained evaluation data can also be used in a more prospective way to derive useful hints for better dialogue design (prospective correlation analysis). All the presented conclusions are systematically illustrated by the concrete results obtained during the experiments that have been carried out in the InfoVox project.

Full Paper

Bibliographic reference.  Rajman, Martin / Rajman, Andréa / Seydoux, Florian / Trutnev, Alex (2003): "Assessing the usability of a dialogue management system designed in the framework of a rapid dialogue prototyping methodology", In AQS-2003, 126-133.