Automatic detection of disfluencies in spoken language is important for making speech recognition output more readable, and for aiding downstream language processing modules. We compare a generative hidden Markov model (HMM)-based approach and two conditional models - a maximum entropy (Maxent) model and a conditional random field (CRF) - for detecting disfluencies in speech. The conditional modeling approaches provide a more principled way to model correlated features. In particular, the CRF approach directly detects the reparandum regions, and thus avoids the use of ad-hoc heuristic rules. We evaluate performance of these three models across two different corpora (conversational speech and broadcast news) and for two types of transcriptions (human transcriptions and recognition output). Overall we find that the conditional modeling approaches (Maxent and CRF) tend to outperform (with one exception) the HMM approach. Effects of speaking style, word recognition errors, and future directions are also discussed.
Cite as: Liu, Y., Shriberg, E., Stolcke, A., Harper, M. (2005) Comparing HMM, maximum entropy, and conditional random fields for disfluency detection. Proc. Interspeech 2005, 3313-3316, doi: 10.21437/Interspeech.2005-851
@inproceedings{liu05f_interspeech,
author={Yang Liu and Elizabeth Shriberg and Andreas Stolcke and Mary Harper},
title={{Comparing HMM, maximum entropy, and conditional random fields for disfluency detection}},
year=2005,
booktitle={Proc. Interspeech 2005},
pages={3313--3316},
doi={10.21437/Interspeech.2005-851}
}