Auditory-Visual Speech Processing 2005

British Columbia, Canada
July 24-27, 2005

Improved Speech Reading Through a Free-Parts Representation

Simon Lucey (1), Patrick Lucey (2)

(1) Advanced Multimedia Processing Laboratory, Department of Electrical and Computer Engineering, Carnegie Mellon University, Pittsburgh, PA, USA
(2) Speech, Audio, Image and Video Research Laboratory, Queensland University of Technology, Brisbane, Australia

Motivated by the success of free-parts based representations in face recognition [1] we have attempted to address some of the problems associated with applying such a philosophy to the task of speaker-independent automatic speech reading. Hitherto, a major problem with canonical area-based approaches in automatic speech reading is the intrinsic lack of training observations due to the visual speech modality's low sample rate and large variability in appearance. We believe a free-parts representation can overcome many of these limitations due to its natural ability to generalize by producing many observations from a single mouth image, whilst still preserving the ability to discriminate between various visual-speech units. This approach additionally requires a modification to traditional techniques employed for the estimation of hidden Markov Models (HMMs), whose resultant models we currently refer to as free-parts HMMs (FP-HMMs). Results will be presented on the CUAVE audio-visual speech database.

Full Paper

Bibliographic reference.  Lucey, Simon / Lucey, Patrick (2005): "Improved speech reading through a free-parts representation", In AVSP-2005, 85-86.