ISCA Archive Interspeech 2008
ISCA Archive Interspeech 2008

Speech-driven lip motion generation with a trajectory HMM

Gregor Hofer, Junichi Yamagishi, Hiroshi Shimodaira

Automatic speech animation remains a challenging problem that can be described as finding the optimal sequence of animation parameter configurations given some speech. In this paper we present a novel technique to automatically synthesise lip motion trajectories from a speech signal. The developed system predicts lip motion units from the speech signal and generates animation trajectories automatically employing a "Trajectory Hidden Markov Model". Using the MLE criterion, its parameter generation algorithm produces the optimal smooth motion trajectories that are used to drive control points on the lips directly. Additionally, experiments were carried out to find a suitable model unit that produces the most accurate results. Finally a perceptual evaluation was conducted, that showed that the developed motion units perform better than phonemes.


doi: 10.21437/Interspeech.2008-591

Cite as: Hofer, G., Yamagishi, J., Shimodaira, H. (2008) Speech-driven lip motion generation with a trajectory HMM. Proc. Interspeech 2008, 2314-2317, doi: 10.21437/Interspeech.2008-591

@inproceedings{hofer08_interspeech,
  author={Gregor Hofer and Junichi Yamagishi and Hiroshi Shimodaira},
  title={{Speech-driven lip motion generation with a trajectory HMM}},
  year=2008,
  booktitle={Proc. Interspeech 2008},
  pages={2314--2317},
  doi={10.21437/Interspeech.2008-591}
}