Auditory-Visual Speech Processing (AVSP) 2010

Hakone, Kanagawa, Japan
September 30-October 3, 2010

Evaluation of Real-time Audio-Visual Speech Recognition

Peng Shen (1), Satoshi Tamura (2), Satoru Hayamizu (2)

(1) Graduate School of Engineering, Gifu University, Japan
(2) Faculty of Engineering, Gifu University, Japan

In this paper, we propose and develop a real-time audio-visual automatic continuous speech recognition system. The system utilizes live speech signals and facial images that collected from a microphone and a camera. Optical-flow-based features are used as visual feature. VAD technology and lip tracking are utilized to improve recognition accuracy. In this paper, several experiments are conducted using Japanese connected digit speech contaminated with white noise, music, television news and car engine noise. Experimental results show when the user is listening news or in a running car with window open the recognition accuracy of the proposed system are not enough. The accuracy of the proposed system is high at a place with light music or in a running car with window close.

Index Terms: speech recognition system, multi-modal, real-time, optical-flow

Full Paper

Bibliographic reference.  Shen, Peng / Tamura, Satoshi / Hayamizu, Satoru (2010): "Evaluation of real-time audio-visual speech recognition", In AVSP-2010, paper P4.