Auditory-Visual Speech Processing 2007 (AVSP2007)

Kasteel Groenendaal, Hilvarenbeek, The Netherlands
August 31 - September 3, 2007

Consequences on bimodal perception of the timing of the consonant and vowel audiovisual flows

Emilie Troille (1,2), Marie-Agnès Cathiard (2), Christian Abry (2)

(1) GIPSA-Lab-ICP; (2) Université Stendhal, Grenoble, France

It is now well known that speech can be seen before it is heard: this has been extensively shown for the vowel rounding anticipatory gesture leading the sound. In this study, the perception of French vowel [y] anticipatory coarticulation was tested throughout a voiced fricative [z] consonant with a gating paradigm. It was found that vowel auditory information, as carried by the noise of the fricative, was ahead of visual and even audiovisual information. Hence the time course of bimodal information in speech cannot be considered to display the same pattern whatever the timing of the coordination of speech gestures. As concerns vowel information only, consonantal coarticulation can carry earlier auditory information than the vowel itself, this depending of the structure of the stimulus. In our fricative-vowel case, it was obvious that the vowel building up movement was audible throughout the fricative noise, whereas the changes in formant grouping (indicating mainly the change from [i] to [y]) occurred later. As concerns the timing of audio and visual information, it becomes more and more obvious that the dates of the delivery of each component of the perceptual flow can change drastically the results obtained in the time course of brain activity for bimodal processing. Hence the flexibility of the phase relationships between the two flows urges to a better knowledge of the natural ecological variability in audiovisual signal production, a flexible coherence not to be violated on pain of breaking the Gestalt laws.

Full Paper

Bibliographic reference.  Troille, Emilie / Cathiard, Marie-Agnès / Abry, Christian (2007): "Consequences on bimodal perception of the timing of the consonant and vowel audiovisual flows", In AVSP-2007, paper P36.