Multilingual Neural Network Acoustic Modelling for ASR of Under-Resourced English-isiZulu Code-Switched Speech

Astik Biswas, Febe de Wet, Ewald van der Westhuizen, Emre Yılmaz, Thomas Niesler


Although isiZulu speakers code-switch with English as a matter of course, extremely little appropriate data is available for acoustic modelling. Recently, a small five-language corpus of code-switched South African soap opera speech was compiled. We used this corpus to evaluate the application of multilingual neural network acoustic modelling to English-isiZulu code-switched speech recognition. Our aim was to determine whether English-isiZulu speech recognition accuracy can be improved by incorporating three other language pairs in the corpus: English-isiXhosa, English-Setswana and English-Sesotho. Since isiXhosa, like isiZulu, belongs to the Nguni language family, while Setswana and Sesotho belong to the more distant Sotho family, we could also investigate the merits of additional data from within and across language groups. Our experiments using both fully connected DNN and TDNN-LSTM architectures show that English-isiZulu speech recognition accuracy as well as language identification after code-switching is improved more by the incorporation of English-isiXhosa data than by the incorporation of the other language pairs. However additional data from the more distant language group remained beneficial and the best overall performance was always achieved with a multilingual neural network trained on all four language pairs.


 DOI: 10.21437/Interspeech.2018-1711

Cite as: Biswas, A., de Wet, F., van der Westhuizen, E., Yılmaz, E., Niesler, T. (2018) Multilingual Neural Network Acoustic Modelling for ASR of Under-Resourced English-isiZulu Code-Switched Speech. Proc. Interspeech 2018, 2603-2607, DOI: 10.21437/Interspeech.2018-1711.


@inproceedings{Biswas2018,
  author={Astik Biswas and Febe {de Wet} and Ewald {van der Westhuizen} and Emre Yılmaz and Thomas Niesler},
  title={Multilingual Neural Network Acoustic Modelling for ASR of Under-Resourced English-isiZulu Code-Switched Speech},
  year=2018,
  booktitle={Proc. Interspeech 2018},
  pages={2603--2607},
  doi={10.21437/Interspeech.2018-1711},
  url={http://dx.doi.org/10.21437/Interspeech.2018-1711}
}