9th Annual Conference of the International Speech Communication Association

Brisbane, Australia
September 22-26, 2008

Speech-Driven Lip Motion Generation with a Trajectory HMM

Gregor Hofer, Junichi Yamagishi, Hiroshi Shimodaira

University of Edinburgh, UK

Automatic speech animation remains a challenging problem that can be described as finding the optimal sequence of animation parameter configurations given some speech. In this paper we present a novel technique to automatically synthesise lip motion trajectories from a speech signal. The developed system predicts lip motion units from the speech signal and generates animation trajectories automatically employing a "Trajectory Hidden Markov Model". Using the MLE criterion, its parameter generation algorithm produces the optimal smooth motion trajectories that are used to drive control points on the lips directly. Additionally, experiments were carried out to find a suitable model unit that produces the most accurate results. Finally a perceptual evaluation was conducted, that showed that the developed motion units perform better than phonemes.

Full Paper

Bibliographic reference.  Hofer, Gregor / Yamagishi, Junichi / Shimodaira, Hiroshi (2008): "Speech-driven lip motion generation with a trajectory HMM", In INTERSPEECH-2008, 2314-2317.