12th Annual Conference of the International Speech Communication Association

Florence, Italy
August 27-31. 2011

Speech Transcript Evaluation for Information Retrieval

Laurens van der Werff (1), Wessel Kraaij (2), Franciska de Jong (1)

(1) University of Twente, The Netherlands
(2) Radboud Universiteit Nijmegen, The Netherlands

Speech recognition transcripts are being used in various fields of research and practical applications, putting various demands on their accuracy. Traditionally ASR research has used intrinsic evaluation measures such as word error rate to determine transcript quality. In non-dictation-type applications such as speech retrieval, it is better to use extrinsic (or task specific) measures. Indexation and the associated processing may eliminate certain errors, whereas the search query may reveal others. In this work, we argue that the standard extrinsic speech retrieval measure average precision is unpractical for ASR evaluation. As an alternative we propose the use of ranked correlation measures on the output of the speech retrieval task, with the goal of predicting relative mean average precision. The measures we used showed a reasonably high correlation with average precision, but require much less human effort to calculate and can be more easily deployed in a variety of real-life settings.

Full Paper

Bibliographic reference.  Werff, Laurens van der / Kraaij, Wessel / Jong, Franciska de (2011): "Speech transcript evaluation for information retrieval", In INTERSPEECH-2011, 1525-1528.