2003 ISCA Workshop on Multilingual Spoken Document Retrieval

Hong Kong
April 4-5, 2003

Evaluation of Extractive Voicemail Summarization

Konstantinos Koumpis, Steve Renals

Department of Computer Science, University of Sheffield, UK

This paper is about the evaluation of a system that generates short text summaries of voicemail messages, suitable for transmission as text messages. Our approach to summarization is based on a speech-recognized transcript of the voicemail message, from which a set of summary words is extracted. The system uses a classiŞer to identify the summary words, with each word being identiŞed by a vector of lexical and prosodic features. The features are selected using Parcel, an ROC-based algorithm. Our evaluations of the system, using a slot error rate metric, have compared manual and automatic summarization, and manual and automatic recognition (using two different recognizers). We also report on two subjective evaluations using mean opinion score of summaries, and a set of comprehension tests. The main results from these experiments were that the perceived difference in quality of summarization was affected more by errors resulting from automatic transcription, than by the automatic summarization process.

Full Paper

Bibliographic reference.  Koumpis, Konstantinos / Renals, Steve (2003): "Evaluation of extractive voicemail summarization", In MSDR-2003, 19-24.