15th Annual Conference of the International Speech Communication Association

September 14-18, 2014

Language Modeling with Sum-Product Networks

Wei-Chen Cheng (1), Stanley Kok (1), Hoai Vu Pham (1), Hai Leong Chieu (2), Kian Ming A. Chai (2)

(1) SUTD, Singapore
(2) DSO, Singapore

Sum product networks (SPNs) are a new class of deep probabilistic models. They can contain multiple hidden layers while keeping their inference and training times tractable. An SPN consists of interleaving layers of sum nodes and product nodes. A sum node can be interpreted as a hidden variable, and a product node can be viewed as a feature capturing rich interactions among an SPN's inputs. We show that the ability of SPN to use hidden layers to model complex dependencies among words, and its tractable inference and learning times, make it a suitable framework for a language model. Even though SPNs have been applied to a variety of vision problems, we are the first to use it for language modeling. Our empirical comparisons with six previous language models indicate that our SPN has superior performance.

Full Paper

Bibliographic reference.  Cheng, Wei-Chen / Kok, Stanley / Pham, Hoai Vu / Chieu, Hai Leong / Chai, Kian Ming A. (2014): "Language modeling with sum-product networks", In INTERSPEECH-2014, 2098-2102.