Comparative Study of Parametric and Representation Uncertainty Modeling for Recurrent Neural Network Language Models

Jianwei Yu, Max W.Y. Lam, Shoukang Hu, Xixin Wu, Xu Li, Yuewen Cao, Xunying Liu, Helen Meng


Recurrent neural network language models (RNNLMs) have shown superior performance across a range of tasks, including speech recognition. The hidden layer of RNNLMs plays a vital role in learning the suitable representation of contexts for word prediction. However, the deterministic model parameters and fixed hidden vectors in conventional RNNLMs have limited power in modeling the uncertainty over hidden representations. In order to address this issue, in this paper, a comparative study of parametric and hidden representation uncertainty modeling approaches based on Bayesian gates and variational RNNLMs respectively is investigated on long short-term memory (LSTM) and gated recurrent units (GRU) LMs. Experimental results are presented on two tasks: PennTreebank (PTB) corpus, Switchboard conversational telephone speech (SWBD). Consistent performance improvements were obtained over conventional RNNLMs in terms of both perplexity and word error rate.


 DOI: 10.21437/Interspeech.2019-1927

Cite as: Yu, J., Lam, M.W., Hu, S., Wu, X., Li, X., Cao, Y., Liu, X., Meng, H. (2019) Comparative Study of Parametric and Representation Uncertainty Modeling for Recurrent Neural Network Language Models. Proc. Interspeech 2019, 3510-3514, DOI: 10.21437/Interspeech.2019-1927.


@inproceedings{Yu2019,
  author={Jianwei Yu and Max W.Y. Lam and Shoukang Hu and Xixin Wu and Xu Li and Yuewen Cao and Xunying Liu and Helen Meng},
  title={{Comparative Study of Parametric and Representation Uncertainty Modeling for Recurrent Neural Network Language Models}},
  year=2019,
  booktitle={Proc. Interspeech 2019},
  pages={3510--3514},
  doi={10.21437/Interspeech.2019-1927},
  url={http://dx.doi.org/10.21437/Interspeech.2019-1927}
}