7th International Conference on Spoken Language Processing

September 16-20, 2002
Denver, Colorado, USA

Audio-Visual Continuous Speech Recognition Using a Coupled Hidden Markov Model

Xiaoxing Liu (1), Yibao Zhao (1), Xiaobo Pi (1), Luhong Liang (1), Ara V. Nefian (2)

(1) Intel China Ltd., China; (2) Intel Corporation, USA

With the increase in the computational complexity of recent computers, audio-visual speech recognition (AVSR) became an attractive research topic that can lead to a robust solution for speech recognition in noisy environments. In the audio visual continuous speech recognition system presented in this paper, the audio and visual observation sequences are integrated using a coupled hidden Markov model (CHMM). The statistical properties of the CHMM can describe the asyncrony of the audio and visual features while preserving their natural correlation over time. The experimental results show that the current system tested on the XM2VTS database reduces the error rate of the audio only speech recognition system at SNR of 0db by over 55%.

Full Paper

Bibliographic reference.  Liu, Xiaoxing / Zhao, Yibao / Pi, Xiaobo / Liang, Luhong / Nefian, Ara V. (2002): "Audio-visual continuous speech recognition using a coupled hidden Markov model", In ICSLP-2002, 213-216.