Auditory-Visual Speech Processing
(AVSP 2001)

September 7-9, 2001
Aalborg, Denmark

Multimodal Translation

Shigeo Morishima (1,2), Shin Ogata (1,2), Satoshi Nakamura (2)

(1) Faculty of Engineering, Seikei University, Tokyo, Japan
(2) ATR, Seika-cho, Souraku-gun, Kyoto, Japan

A stand-in is a common technique for movies and TV programs in foreign languages. The current stand-in that only substitutes the voice channel results awkward matching to the mouth motion. Videophone with automatic voice translation are expected to be widely used in the near future, which may face the same problem without lip-synchronized speaking face image translation. We introduce a multi-modal English-to-Japanese and Japanese-to-English translation system that also translates the speaker's speech motion while synchronizing it to the translated speech. To retain the speaker's facial expression, we substitute only the speech organ's image with the synthesized one, which is made by a three-dimensional wire-frame model that is adaptable to any speaker. Our approach enables image synthesis and translation with an extremely small database. Also, we propose a method to track motion of the face from the video image. In this system, movement and rotation of the head is detected by template matching using a 3D personal face wire-frame model. By this technique, an automatic multimodal translation can be achieved.

Bibliographic reference.  Morishima, Shigeo / Ogata, Shin / Nakamura, Satoshi (2001): "Multimodal translation", In AVSP-2001, 98-103.

Multimedia Files

Link Original Filename Description Format
av01_098_1.mpg (2147 KB) MorishimaDemo.mpg An original and translated video sequence Video File - MPEG
av01_098_2.mpg (2401 KB) MorishimaMaking.mpg The result of face tracking and replaced face with synthetic mouth. Video File - MPEG