Auditory-Visual Speech Processing 2007 (AVSP2007)

Kasteel Groenendaal, Hilvarenbeek, The Netherlands
August 31 - September 3, 2007

The Auditory and the Visual Percept Evoked by the Same Audiovisual Vowels

Hartmut Traunmüller, Niklas Öhrström

Department of Linguistics, University of Stockholm, Sweden

In analyses and models of audiovisual speech perception, it has been common to consider three percepts: (1) the auditory percept evoked by acoustic stimuli, (2) the visual percept evoked by optic stimuli and (3) a common percept evoked by synchronous optic and acoustic stimuli. Here, it is shown that a vocal percept that is heard and influenced by vision has to be distinguished from a gestural percept that is seen and influenced by audition. In the two experiments reported, syllables distinguished solely by their vowels [i], [y] or [e] were presented to phonetically sophisticated subjects auditorily, visually and in incongruently cross-dubbed audiovisual form. In the first, the subjects rated roundedness, lip spreading, openness and backness of the vowels they heard - in the second of those they saw. The results confirmed that roundedness is mainly heard by eye while openness is heard by ear. Heard backness (retraction) varied with the acoustic and optic presence of roundedness. Seen openness was substantially influenced by acoustic cues, while there was no such influence on seen roundedness. The results are discussed in the context of theories and models.

Full Paper

Bibliographic reference.  Traunmüller, Hartmut / Öhrström, Niklas (2007): "The auditory and the visual percept evoked by the same audiovisual vowels", In AVSP-2007, paper L4-1.