ISCA Archive Interspeech 2021
ISCA Archive Interspeech 2021

On Sampling-Based Training Criteria for Neural Language Modeling

Yingbo Gao, David Thulke, Alexander Gerstenberger, Khoa Viet Tran, Ralf Schlüter, Hermann Ney

As the vocabulary size of modern word-based language models becomes ever larger, many sampling-based training criteria are proposed and investigated. The essence of these sampling methods is that the softmax-related traversal over the entire vocabulary can be simplified, giving speedups compared to the baseline. A problem we notice about the current landscape of such sampling methods is the lack of a systematic comparison and some myths about preferring one over another. In this work, we consider Monte Carlo sampling, importance sampling, a novel method we call compensated partial summation, and noise contrastive estimation. Linking back to the three traditional criteria, namely mean squared error, binary cross-entropy, and cross-entropy, we derive the theoretical solutions to the training problems. Contrary to some common belief, we show that all these sampling methods can perform equally well, as long as we correct for the intended class posterior probabilities. Experimental results in language modeling and automatic speech recognition on Switchboard and LibriSpeech support our claim, with all sampling-based methods showing similar perplexities and word error rates while giving the expected speedups.


doi: 10.21437/Interspeech.2021-1067

Cite as: Gao, Y., Thulke, D., Gerstenberger, A., Tran, K.V., Schlüter, R., Ney, H. (2021) On Sampling-Based Training Criteria for Neural Language Modeling. Proc. Interspeech 2021, 1877-1881, doi: 10.21437/Interspeech.2021-1067

@inproceedings{gao21b_interspeech,
  author={Yingbo Gao and David Thulke and Alexander Gerstenberger and Khoa Viet Tran and Ralf Schlüter and Hermann Ney},
  title={{On Sampling-Based Training Criteria for Neural Language Modeling}},
  year=2021,
  booktitle={Proc. Interspeech 2021},
  pages={1877--1881},
  doi={10.21437/Interspeech.2021-1067}
}