Auditory-Visual Speech Processing (AVSP) 2011

Volterra, Italy
September 1-2, 2011

Binding and Unbinding the McGurk Effect in Audiovisual Speech Fusion: Follow-Up Experiments on a New Paradigm

Olha Nahorna, Frédéric Berthommier, Jean-Luc Schwartz

GIPSA-Lab – DPC, ICP, UMR 5216–CNRS Université de Grenoble, France

The McGurk effect demonstrates the existence of a fusion process in audiovisual speech perception: the combination of the sound "ba" with the face of a speaker who pronounces "ga" is frequently perceived as "da". We assume that in the upstream of this phonetic fusion process, there is a “binding” process, which controls the combination of image and sound, and can block or reduce it in the case of audiovisual incoherencies (conditional binding process), as in the case of a dubbed film. To test and explore this binding hypothesis, we designed various experiments in which a coherent or incoherent audiovisual context is placed before McGurk stimuli, and we show that the incoherent contextual stimulus can significantly reduce the McGurk effect.

Index Terms. McGurk effect, binding, multisensory fusion, audiovisual speech perception, audiovisual scene analysis

Full Paper

Bibliographic reference.  Nahorna, Olha / Berthommier, Frédéric / Schwartz, Jean-Luc (2011): "Binding and unbinding the Mcgurk effect in audiovisual speech fusion: follow-up experiments on a new paradigm", In AVSP-2011, 21-24.