12th Annual Conference of the International Speech Communication Association

Florence, Italy
August 27-31. 2011

Predicting Human Perceived Accuracy of ASR Systems

Taniya Mishra, Andrej Ljolje, Mazin Gilbert

AT&T Labs Research, USA

Word error rate (WER), which is the most commonly used method of measuring automatic speech recognition (ASR) accuracy, penalizes all types of ASR errors equally. However, humans differentially weigh different types of ASR errors. They judge ASR errors that distort the meaning of the spoken message more harshly than those that do not. Aiming to align more closely with human perception of ASR accuracy, we developed a new metric HPA (Human Perceived Accuracy) that predicts the subjective perceived accuracy of ASR transcriptions. HPA is computed based on the central idea of differential weighting of different ASR errors. Applied to the particular task of automatically recognizing voicemails, we found that the correlation between HPA and the human judgement of ASR accuracy was significantly higher (r-value=0.91) than the correlation between WER and human judgement (r-value=0.65).

Full Paper

Bibliographic reference.  Mishra, Taniya / Ljolje, Andrej / Gilbert, Mazin (2011): "Predicting human perceived accuracy of ASR systems", In INTERSPEECH-2011, 1945-1948.