Interspeech'2005 - Eurospeech

Lisbon, Portugal
September 4-8, 2005

A New Posterior Based Audio-Visual Integration Method for Robust Speech Recognition

Rowan Seymour, Ji Ming, Darryl Stewart

Queen's University Belfast, UK

We describe the development of a multistream HMM based audiovisual speech recognition (AVSR) system and a new method for integrating the audio and visual streams using frame level posterior probabilities. This is compared to the standard feature concatenation and weighted product methods in speaker-dependent tests using our own multimodal database, by examining speech recognition robustness to corruption in either stream. For corruption in the audio stream we use additive noise at different SNR levels, and for corruption in the video stream we use MPEG4 compression at different bitrates as well as image blurring using Gaussian filters. We provide very promising results which demonstrate the robustness of the new method.

Full Paper

Bibliographic reference.  Seymour, Rowan / Ming, Ji / Stewart, Darryl (2005): "A new posterior based audio-visual integration method for robust speech recognition", In INTERSPEECH-2005, 1229-1232.