Sixth International Conference on Spoken Language Processing
(ICSLP 2000)

Beijing, China
October 16-20, 2000

Web-Based Monitoring, Logging and Reporting Tools for Multi-Service Multi-Modal Systems

Giuseppe Di Fabbrizio, Shrikanth Narayanan

AT&T Labs - Research, Florham Park, NJ, USA

This paper describes MILER (Multi-modal data Logger for Evaluation and Report), a web-based multi-service monitoring, logging and reporting tool for advanced multimodal dialog systems. MILER has been designed to directly arrange and synchronize logging data collected from live services and to provide real-time reports about service usage and system performance. Special attention has been given to the architecture design in order to achieve service and access-device independence and reliable synchronization of data from distributed logs. MILER allows researchers to analyze multi-modal interactions, analyze the call flow, reconstruct the system/user dialogue turns, play the recorded user utterances, and provide a preliminary dialogue performance evaluation. It also supports labeling and annotation of the dialogue turns for further offline analysis. Once the user inputs (i.e. speech and other input modalities) are manually transcribed and labeled, along with detailed log events from each dialog, MILER derives a set of objective measures, which includes word and concept accuracy, number of attempts per concept, dialog turn counts and duration, and task completion rates. Subjective measures extracted from user's surveys, including perceived task success and ease of use measures, can be combined with the objective measures and the results used later for accuracy computation.

Full Paper

Bibliographic reference.  Fabbrizio, Giuseppe Di / Narayanan, Shrikanth (2000): "Web-based monitoring, logging and reporting tools for multi-service multi-modal systems", In ICSLP-2000, vol.4, 736-739.