15th Annual Conference of the International Speech Communication Association

September 14-18, 2014

Comparing Approaches to Convert Recurrent Neural Networks into Backoff Language Models for Efficient Decoding

Heike Adel (1), Katrin Kirchhoff (2), Ngoc Thang Vu (1), Dominic Telaar (1), Tanja Schultz (1)

(1) KIT, Germany
(2) University of Washington, USA

In this paper, we investigate and compare three different possibilities to convert recurrent neural network language models (RNNLMs) into backoff language models (BNLM). While RNNLMs often outperform traditional n-gram approaches in the task of language modeling, their computational demands make them unsuitable for an efficient usage during decoding in an LVCSR system. It is, therefore, of interest to convert them into BNLMs in order to integrate their information into the decoding process. This paper compares three different approaches: a text based conversion, a probability based conversion and an iterative conversion. The resulting language models are evaluated in terms of perplexity and mixed error rate in the context of the Code-Switching data corpus SEAME. Although the best results are obtained by combining the results of all three approaches, the text based conversion approach alone leads to significant improvements on the SEAME corpus as well while offering the highest computational efficiency. In total, the perplexity can be reduced by 11.4% relative on the evaluation set and the mixed error rate by 3.0% relative on the same data set.

Full Paper

Bibliographic reference.  Adel, Heike / Kirchhoff, Katrin / Vu, Ngoc Thang / Telaar, Dominic / Schultz, Tanja (2014): "Comparing approaches to convert recurrent neural networks into backoff language models for efficient decoding", In INTERSPEECH-2014, 651-655.