ISCA Archive Interspeech 2013
ISCA Archive Interspeech 2013

Exploiting the succeeding words in recurrent neural network language models

Yangyang Shi, Martha Larson, Pascal Wiggers, Catholijn M. Jonker

In automatic speech recognition, conventional language models recognize the current word using only information from preceding words. Recently, Recurrent Neural Network Language Models (RNNLMs) have drawn increased research attention because of their ability to outperform conventional n-gram language models. The superiority of RNNLMs is based in their ability to capture longdistance word dependencies. RNNLMs are, in practice, applied in an N-best rescoring framework, which offers new possibilities for information integration. In particular, it becomes interesting to extend the ability of RNNLMs to capture long distance information by also allowing them to exploit information from succeeding words during the rescoring process. This paper proposes three approaches for exploiting succeeding word information in RNNLMs. The first is a forward-backward model that combines RNNLMs exploiting preceding and succeeding words. The second is an extension of a Maximum Entropy RNNLM (RNNME) that incorporates succeeding word information. The third is an approach that combines language models using two-pass alternating rescoring. Experimental results demonstrate the ability of succeeding word information to improve RNNLM performance, both in terms of perplexity and Word Error Rate (WER). The best performance is achieved by a combined model that exploits the three words succeeding the current word.


doi: 10.21437/Interspeech.2013-183

Cite as: Shi, Y., Larson, M., Wiggers, P., Jonker, C.M. (2013) Exploiting the succeeding words in recurrent neural network language models. Proc. Interspeech 2013, 632-636, doi: 10.21437/Interspeech.2013-183

@inproceedings{shi13_interspeech,
  author={Yangyang Shi and Martha Larson and Pascal Wiggers and Catholijn M. Jonker},
  title={{Exploiting the succeeding words in recurrent neural network language models}},
  year=2013,
  booktitle={Proc. Interspeech 2013},
  pages={632--636},
  doi={10.21437/Interspeech.2013-183}
}