ISCA Archive Eurospeech 1999
ISCA Archive Eurospeech 1999

A missing-word test comparison of human and statistical language model performance

Marie Owens, Anja Krüger, Paul Donnelly, F J Smith, Ji Ming

A suite of missing-word tests based on text extracts selected randomly from two different text corpora provided a metric which was used in an evaluation of human performance, an evaluation of language model performance and a cross-comparison of the performances. The effects of providing different sizes of context for the missing word (ranging from two words to three sentences) were examined and two main patterns became clear from the results: - surprisingly, for tests where the language model was able to take advantage of all the context information provided (i.e. where the context consisted of just a few words) it outperformed humans; - conversely, humans outperformed the language model when the size of context given for the missing word exceeded the size, which the language model could usefully, employ in its probability calculations (typically more than six words).


doi: 10.21437/Eurospeech.1999-40

Cite as: Owens, M., Krüger, A., Donnelly, P., Smith, F.J., Ming, J. (1999) A missing-word test comparison of human and statistical language model performance. Proc. 6th European Conference on Speech Communication and Technology (Eurospeech 1999), 145-148, doi: 10.21437/Eurospeech.1999-40

@inproceedings{owens99_eurospeech,
  author={Marie Owens and Anja Krüger and Paul Donnelly and F J Smith and Ji Ming},
  title={{A missing-word test comparison of human and statistical language model performance}},
  year=1999,
  booktitle={Proc. 6th European Conference on Speech Communication and Technology (Eurospeech 1999)},
  pages={145--148},
  doi={10.21437/Eurospeech.1999-40}
}