ISCA Archive S4SG 2022
ISCA Archive S4SG 2022

Assessing ASR Model Quality on Disordered Speech using BERTScore

Jimmy Tobin, Qisheng Li, Subhashini Venugopalan, Katie Seaver, Richard Cave, Katrin Tomanek

Word Error Rate (WER) is the primary metric used to assess automatic speech recognition (ASR) model quality. It has been shown that ASR models tend to have much higher WER on speakers with speech impairments than typical English speakers. It is hard to determine if models can be be useful at such high error rates. This study investigates the use of BERTScore, an evaluation metric for text generation, to provide a more informative measure of ASR model quality and usefulness. Both BERTScore and WER were compared to prediction errors manually annotated by Speech Language Pathologists for error type and assessment. BERTScore was found to be more correlated with human assessment of error type and assessment. BERTScore was specifically more robust to orthographic changes (contraction and normalization errors) where meaning was preserved. Furthermore, BERTScore was a better fit of error assessment than WER, as measured using an ordinal logistic regression and the Akaike’s Information Criterion (AIC). Overall, our findings suggest that BERTScore can complement WER when assessing ASR model performance from a practical perspective, especially for accessibility applications where models are useful even at lower accuracy than for typical speech.


doi: 10.21437/S4SG.2022-6

Cite as: Tobin, J., Li, Q., Venugopalan, S., Seaver, K., Cave, R., Tomanek, K. (2022) Assessing ASR Model Quality on Disordered Speech using BERTScore. Proc. 1st Workshop on Speech for Social Good (S4SG), 26-30, doi: 10.21437/S4SG.2022-6

@inproceedings{tobin22_s4sg,
  author={Jimmy Tobin and Qisheng Li and Subhashini Venugopalan and Katie Seaver and Richard Cave and Katrin Tomanek},
  title={{Assessing ASR Model Quality on Disordered Speech using BERTScore}},
  year=2022,
  booktitle={Proc. 1st Workshop on Speech for Social Good (S4SG)},
  pages={26--30},
  doi={10.21437/S4SG.2022-6}
}