Auditory-Visual Speech Processing (AVSP 2001) September 7-9, 2001 Aalborg, Denmark

Creating and Controlling Video-Realistic Talking Heads

F. Elisei, M. Odisio, Gérard Bailly, Pierre Badin

Institut de la Communication Parlée, Grenoble, France

We present a linear three-dimensional modeling paradigm for lips and face, that captures the audiovisual speech activity of a given speaker by only six parameters. Our articulatory models are constructed from real data (front and profile images), using a linear component analysis of about 200 3D coordinates of fleshpoints on the subject's face and lips. Compared to a raw component analysis, our construction approach leads to somewhat more comparable relations across subjects: by construction, the six parameters have a clear phonetic/articulatory interpretation. We use such a speaker's specific articulatory model to regularize MPEG-4 facial articulation parameters (FAP) and show that this regularization process can drastically reduce bandwidth, noise and quantization artifacts. We then present how analysis-by-synthesis techniques using the speaker-specific model allows the tracking of facial movements. Finally, the results of this tracking scheme have been used to develop a text-to-audiovisual speech system.

Bibliographic reference.  Elisei, F. / Odisio, M. / Bailly, Gérard / Badin, Pierre (2001): "Creating and controlling video-realistic talking heads", In AVSP-2001, 90-97.

Multimedia Files