ISCA Archive SSW 2021
ISCA Archive SSW 2021

Speech Synthesis from Text and Ultrasound Tongue Image-based Articulatory Input

Tamás Gábor Csapó, László Tóth, Gábor Gosztolya, Alexandra Markó

Articulatory information has been shown to be effective in improving the performance of HMM-based and DNN-based textto- speech synthesis. Speech synthesis research focuses traditionally on text-to-speech conversion, when the input is text or an estimated linguistic representation, and the target is synthesized speech. However, a research field that has risen in the last decade is articulation-to-speech synthesis (with a target application of a Silent Speech Interface, SSI), when the goal is to synthesize speech from some representation of the movement of the articulatory organs. In this paper, we extend traditional (vocoder-based) DNN-TTS with articulatory input, estimated from ultrasound tongue images. We compare text-only, ultrasound-only, and combined inputs. Using data from eight speakers, we show that that the combined text and articulatory input can have advantages in limited-data scenarios, namely, it may increase the naturalness of synthesized speech compared to single text input. Besides, we analyze the ultrasound tongue recordings of several speakers, and show that misalignments in the ultrasound transducer positioning can have a negative effect on the final synthesis performance.


doi: 10.21437/SSW.2021-6

Cite as: Csapó, T.G., Tóth, L., Gosztolya, G., Markó, A. (2021) Speech Synthesis from Text and Ultrasound Tongue Image-based Articulatory Input. Proc. 11th ISCA Speech Synthesis Workshop (SSW 11), 31-36, doi: 10.21437/SSW.2021-6

@inproceedings{csapo21b_ssw,
  author={Tamás Gábor Csapó and László Tóth and Gábor Gosztolya and Alexandra Markó},
  title={{Speech Synthesis from Text and Ultrasound Tongue Image-based Articulatory Input}},
  year=2021,
  booktitle={Proc. 11th ISCA Speech Synthesis Workshop (SSW 11)},
  pages={31--36},
  doi={10.21437/SSW.2021-6}
}