8th European Conference on Speech Communication and Technology

Geneva, Switzerland
September 1-4, 2003


Evaluation Method for Automatic Speech Summarization

Chiori Hori (1), Takaaki Hori (1), Sadaoki Furui (2)

(1) NTT Corporation, Japan
(2) Tokyo Institute of Technology, Japan

We have proposed an automatic speech summarization approach that extracts words from transcription results obtained by automatic speech recognition (ASR) systems. To numerically evaluate this approach, the automatic summarization results are compared with manual summarization generated by humans through word extraction. We have proposed three metrics, weighted word precision, word strings precision and summarization accuracy (series SumACCY), based on a word network created by merging manual summarization results. In this paper, we propose a new metric for automatic summarization results, weighted summarization accuracy (series WSumACCY). This accuracy is weighted by the posterior probability of the manual summaries in the network to give the reliability of each answer extracted from the network. We clarify the goal of each metric and use these metrics to provide automatic evaluation results of the summarized speech. To compare the performance of each evaluation metric, correlations between the evaluation results using these metrics and subjective evaluation by hand are measured. It is confirmed that series WSumACCY is an effective and robust measure for automatic summarization.

Full Paper

Bibliographic reference.  Hori, Chiori / Hori, Takaaki / Furui, Sadaoki (2003): "Evaluation method for automatic speech summarization", In EUROSPEECH-2003, 2825-2828.