ISCA Archive SSW 2007
ISCA Archive SSW 2007

Analysis methods for assessing TTS intelligibility

H. Timothy Bunnell, Jason Lilley

Semantically unpredictable (SU) sentences are often used to assess intelligibility of TTS systems, but analyses of listener responses to SU sentences can be a labor-intensive process. In this paper we compare several approaches to the analysis of data from an SUS task. Data from a study comparing five TTS systems were analyzed in a variety of ways ranging from string edit measures based on carefully hand-corrected phonetically transcribed responses to largely uncorrected words- or sentences-correct measures. Results suggest that a simple sentences-correct measure is adequate when only rank order information is of interest. However, the sentencescorrect measure masks the magnitude of differences between systems and should be avoided when it is important to gage how large the difference in intelligibility is between systems. In preparing response data for analysis, careful human interpretation of listener response data can lead to higher intelligibility measures overall, but does not interact with TTS system or other factors and consequently does not lead to different conclusions when comparing multiple TTS systems. This suggests that largely automated scoring procedures are feasible.

Cite as: Bunnell, H.T., Lilley, J. (2007) Analysis methods for assessing TTS intelligibility. Proc. 6th ISCA Workshop on Speech Synthesis (SSW 6), 374-379

  author={H. Timothy Bunnell and Jason Lilley},
  title={{Analysis methods for assessing TTS intelligibility}},
  booktitle={Proc. 6th ISCA Workshop on Speech Synthesis (SSW 6)},