16th Annual Conference of the International Speech Communication Association

Dresden, Germany
September 6-10, 2015

Objective Intelligibility Assessment of Text-to-Speech Systems Through Utterance Verification

Raphael Ullmann, Ramya Rasipuram, Mathew Magimai-Doss, Hervé Bourlard

Idiap Research Institute, Switzerland

Objective assessment of synthetic speech intelligibility can be a useful tool for the development of text-to-speech (TTS) systems, as it provides a reproducible and inexpensive alternative to subjective listening tests. In a recent work, it was shown that the intelligibility of synthetic speech could be assessed objectively by comparing two sequences of phoneme class conditional probabilities, corresponding to instances of synthetic and human reference speech, respectively. In this paper, we build on those findings to propose a novel approach that formulates objective intelligibility assessment as an utterance verification problem using hidden Markov models, thereby alleviating the need for human reference speech. Specifically, given each text input to the TTS system, the proposed approach automatically verifies the words in the output synthetic speech signal and estimates an intelligibility score based on word recall statistics. We evaluate the proposed approach on the 2011 Blizzard Challenge data, and show that the estimated scores and the subjective intelligibility scores are highly correlated (Pearson's |R| = 0.94).

Full Paper

Bibliographic reference.  Ullmann, Raphael / Rasipuram, Ramya / Magimai-Doss, Mathew / Bourlard, Hervé (2015): "Objective intelligibility assessment of text-to-speech systems through utterance verification", In INTERSPEECH-2015, 3501-3505.