16th Annual Conference of the International Speech Communication Association

Dresden, Germany
September 6-10, 2015

Latent Words Recurrent Neural Network Language Models

Ryo Masumura (1), Taichi Asami (1), Takanobu Oba (1), Hirokazu Masataki (1), Sumitaka Sakauchi (1), Akinori Ito (2)

(1) NTT Corporation, Japan
(2) Tohoku University, Japan

This paper proposes a novel language modeling approach called latent word recurrent neural network language model, which solves the problems present in both recurrent neural network language models (RNNLMs) and latent word language models (LWLMs). The proposed model has a soft class structure based on a latent variable space as well as LWLM, where the latent variable space is modeled using RNNLM. From the viewpoint of RNNLMs, the proposed model can be considered as a soft class RNNLM with a vast latent variable space. In contrast, from the viewpoint of LWLMs, the proposed model can be considered as an LWLM that uses the RNN structure for latent variable modeling instead of the n-gram structure. This paper also details the parameter inference method and two kinds of usages for natural language processing tasks. Our experiments show effectiveness of the proposed model on a perplexity evaluation for the Penn Treebank corpus and an automatic speech recognition evaluation for Japanese spontaneous speech tasks.

Full Paper

Bibliographic reference.  Masumura, Ryo / Asami, Taichi / Oba, Takanobu / Masataki, Hirokazu / Sakauchi, Sumitaka / Ito, Akinori (2015): "Latent words recurrent neural network language models", In INTERSPEECH-2015, 2380-2384.