Speech Model Pre-Training for End-to-End Spoken Language Understanding

Loren Lugosch, Mirco Ravanelli, Patrick Ignoto, Vikrant Singh Tomar, Yoshua Bengio


Whereas conventional spoken language understanding (SLU) systems map speech to text, and then text to intent, end-to-end SLU systems map speech directly to intent through a single trainable model. Achieving high accuracy with these end-to-end models without a large amount of training data is difficult. We propose a method to reduce the data requirements of end-to-end SLU in which the model is first pre-trained to predict words and phonemes, thus learning good features for SLU. We introduce a new SLU dataset, Fluent Speech Commands, and show that our method improves performance both when the full dataset is used for training and when only a small subset is used. We also describe preliminary experiments to gauge the model’s ability to generalize to new phrases not heard during training.


 DOI: 10.21437/Interspeech.2019-2396

Cite as: Lugosch, L., Ravanelli, M., Ignoto, P., Tomar, V.S., Bengio, Y. (2019) Speech Model Pre-Training for End-to-End Spoken Language Understanding. Proc. Interspeech 2019, 814-818, DOI: 10.21437/Interspeech.2019-2396.


@inproceedings{Lugosch2019,
  author={Loren Lugosch and Mirco Ravanelli and Patrick Ignoto and Vikrant Singh Tomar and Yoshua Bengio},
  title={{Speech Model Pre-Training for End-to-End Spoken Language Understanding}},
  year=2019,
  booktitle={Proc. Interspeech 2019},
  pages={814--818},
  doi={10.21437/Interspeech.2019-2396},
  url={http://dx.doi.org/10.21437/Interspeech.2019-2396}
}