International Conference on Auditory-Visual Speech Processing 2008 Tangalooma Wild Dolphin Resort, Moreton Island, Queensland, Australia September 26-29, 2008

Building a Portable Gesture-to-Audio/Visual Speech System

Sidney Fels (1), Robert Pritchard (2), Eric Vatikiotis-Bateson (3)

(1) Electrical and Computer Engineering, (2) School of Music, (3) Department of Linguistics; University of British Columbia, Vancouver, BC, Canada

We have constructed an easy-to-use portable, wearable gesture-to-speech system based on the Glove-TalkII and GRASSP gesture-controlled speech systems and a vizeme based face-synthesizer. Our new portable system is called a Digital Ventriloquized Actor (DIVA) and refines the use of the formant speech synthesizer. Using a DIVA, a user can speak using hand gestures mapped to both synthetic sound and face using a mapping function that preserves gesture trajectories. By making DIVAs portable and self-contained, speakers can communicate with others in the community and perform in new music/theatre stage productions. DIVA performers also allow us to study the relationship between visible gestures and speech/song production.

Full Paper

Bibliographic reference.  Fels, Sidney / Pritchard, Robert / Vatikiotis-Bateson, Eric (2008): "Building a portable gesture-to-audio/visual speech system", In AVSP-2008, 13-18.