Self-Imitating Feedback Generation Using GAN for Computer-Assisted Pronunciation Training

Seung Hee Yang, Minhwa Chung


Self-imitating feedback is an effective and learner-friendly method for non-native learners in Computer-Assisted Pronunciation Training. Acoustic characteristics in native utterances are extracted and transplanted onto learner’s own speech input, and given back to the learner as a corrective feedback. Previous works focused on speech conversion using prosodic transplantation techniques based on PSOLA algorithm. Motivated by the visual differences found in spectrograms of native and non-native speeches, we investigated applying GAN to generate self-imitating feedback by utilizing generator’ s ability through adversarial training. Because this mapping is highly under-constrained, we also adopt cycle consistency loss to encourage the output to preserve the global structure, which is shared by native and non-native utterances. Trained on 97,200 spectrogram images of short utterances produced by native and non-native speakers of Korean, the generator is able to successfully transform the non-native spectrogram input to a spectrogram with properties of self-imitating feedback. Furthermore, the transformed spectrogram shows segmental corrections that cannot be obtained by prosodic transplantation. Perceptual test comparing the self-imitating and correcting abilities of our method with the baseline PSOLA method shows that the generative approach with cycle consistency loss is promising.


 DOI: 10.21437/Interspeech.2019-1478

Cite as: Yang, S.H., Chung, M. (2019) Self-Imitating Feedback Generation Using GAN for Computer-Assisted Pronunciation Training. Proc. Interspeech 2019, 1881-1885, DOI: 10.21437/Interspeech.2019-1478.


@inproceedings{Yang2019,
  author={Seung Hee Yang and Minhwa Chung},
  title={{Self-Imitating Feedback Generation Using GAN for Computer-Assisted Pronunciation Training}},
  year=2019,
  booktitle={Proc. Interspeech 2019},
  pages={1881--1885},
  doi={10.21437/Interspeech.2019-1478},
  url={http://dx.doi.org/10.21437/Interspeech.2019-1478}
}