 |
2003 ISCA Workshop on
Multilingual Spoken Document Retrieval
(MSDR2003)
Hong Kong
April 4-5, 2003 |
 |
Evaluation of Extractive Voicemail Summarization
Konstantinos Koumpis, Steve Renals
Department of Computer Science, University of Sheffield, UK
This paper is about the evaluation of a system that generates
short text summaries of voicemail messages, suitable
for transmission as text messages. Our approach to
summarization is based on a speech-recognized transcript
of the voicemail message, from which a set of summary
words is extracted. The system uses a classiŞer to identify
the summary words, with each word being identiŞed
by a vector of lexical and prosodic features. The features
are selected using Parcel, an ROC-based algorithm.
Our evaluations of the system, using a slot error rate metric,
have compared manual and automatic summarization,
and manual and automatic recognition (using two
different recognizers). We also report on two subjective
evaluations using mean opinion score of summaries, and
a set of comprehension tests. The main results from these
experiments were that the perceived difference in quality
of summarization was affected more by errors resulting
from automatic transcription, than by the automatic summarization
process.
Full Paper
Bibliographic reference.
Koumpis, Konstantinos / Renals, Steve (2003):
"Evaluation of extractive voicemail summarization",
In MSDR-2003, 19-24.