ISCA Archive Interspeech 2021
ISCA Archive Interspeech 2021

Cross-Lingual Speaker Adaptation Using Domain Adaptation and Speaker Consistency Loss for Text-To-Speech Synthesis

Detai Xin, Yuki Saito, Shinnosuke Takamichi, Tomoki Koriyama, Hiroshi Saruwatari

We present a cross-lingual speaker adaptation method based on domain adaptation and a speaker consistency loss for text-to-speech (TTS) synthesis. Existing monolingual speaker adaptation methods based on direct fine-tuning are not applicable for cross-lingual data. The proposed method first trains a language-independent speaker encoder by speaker verification using domain adaption on multilingual data, including the source and the target languages. Then the proposed method trains a monolingual multi-speaker TTS model on the source language’s data using the speaker embeddings generated by the speaker encoder. To adapt the TTS model of the source language to new speakers the proposed method uses a speaker consistency loss to maximize the cosine similarity between speaker embeddings generated from the natural speech and the same speaker’s synthesized speech. This makes fine-tuning the TTS model of source language on speech data of target language become possible. We conduct experiments on multi-speaker English and Japanese datasets with 207 speakers in total. Results of comprehensive experiments demonstrate that the proposed method can significantly improve speech naturalness compared to the baseline method.


doi: 10.21437/Interspeech.2021-897

Cite as: Xin, D., Saito, Y., Takamichi, S., Koriyama, T., Saruwatari, H. (2021) Cross-Lingual Speaker Adaptation Using Domain Adaptation and Speaker Consistency Loss for Text-To-Speech Synthesis. Proc. Interspeech 2021, 1614-1618, doi: 10.21437/Interspeech.2021-897

@inproceedings{xin21_interspeech,
  author={Detai Xin and Yuki Saito and Shinnosuke Takamichi and Tomoki Koriyama and Hiroshi Saruwatari},
  title={{Cross-Lingual Speaker Adaptation Using Domain Adaptation and Speaker Consistency Loss for Text-To-Speech Synthesis}},
  year=2021,
  booktitle={Proc. Interspeech 2021},
  pages={1614--1618},
  doi={10.21437/Interspeech.2021-897}
}