Auditory-Visual Speech Processing (AVSP) 2009

University of East Anglia, Norwich, UK
September 10-13, 2009

Effective Visually-Derived Wiener Filtering for Audio-Visual Speech Processing

Ibrahim Almajai, Ben Milner

School of Computing Sciences, University of East Anglia, UK

This work presents a novel approach to speech enhancement by exploiting the bimodality of speech and the correlation that exists between audio and visual speech features. For speech enhancement, a visually-derived Wiener filter is developed. This obtains clean speech statistics from visual features by modelling their joint density and making a maximum a posteriori estimate of clean audio from visual speech features. Noise statistics for the Wiener filter utilise an audio-visual voice activity detector which classifies input audio as speech or nonspeech, enabling a noise model to be updated. Analysis shows estimation of speech and noise statistics to be effective with speech quality assessed objectively and subjectively measuring the effectiveness of the resulting Wiener filter. The use of this enhancement method is also considered for ASR purposes.

Index Terms: Audio-visual, speech enhancement, Wiener filter, AVSR, MAP

Full Paper

Bibliographic reference.  Almajai, Ibrahim / Milner, Ben (2009): "Effective visually-derived Wiener filtering for audio-visual speech processing", In AVSP-2009, 134-139.