i-Vectors in Language Modeling: An Efficient Way of Domain Adaptation for Feed-Forward Models

Karel Beneš, Santosh Kesiraju, Lukáš Burget


We show an effective way of adding context information to shallow neural language models. We propose to use Subspace Multinomial Model (SMM) for context modeling and we add the extracted i-vectors in a computationally efficient way. By adding this information, we shrink the gap between shallow feed-forward network and an LSTM from 65 to 31 points of perplexity on the Wikitext-2 corpus (in the case of neural 5-gram model). Furthermore, we show that SMM i-vectors are suitable for domain adaptation and a very small amount of adaptation data (e.g. endmost 5% of a Wikipedia article) brings a substantial improvement. Our proposed changes are compatible with most optimization techniques used for shallow feedforward LMs.


 DOI: 10.21437/Interspeech.2018-1070

Cite as: Beneš, K., Kesiraju, S., Burget, L. (2018) i-Vectors in Language Modeling: An Efficient Way of Domain Adaptation for Feed-Forward Models. Proc. Interspeech 2018, 3383-3387, DOI: 10.21437/Interspeech.2018-1070.


@inproceedings{Beneš2018,
  author={Karel Beneš and Santosh Kesiraju and Lukáš Burget},
  title={i-Vectors in Language Modeling: An Efficient Way of Domain Adaptation for Feed-Forward Models},
  year=2018,
  booktitle={Proc. Interspeech 2018},
  pages={3383--3387},
  doi={10.21437/Interspeech.2018-1070},
  url={http://dx.doi.org/10.21437/Interspeech.2018-1070}
}