ISCA Workshop on Plasticity in Speech Perception (PSP2005)
Senate House, London, UK
This paper will present the results of a study currently in progress in which four related experiments are being used to investigate the mechanisms of early audio-visual integration. The experiments attempt to replicate the findings from Kuhl and Meltzoff's experiment (Kuhl & Meltzoff, 1982) and investigate how far general perceptual mechanisms may account for the observed audiovisual integration in speech. It is hypothesized that the audio-visual integration observed for speech is a particular manifestation of a general propensity to integrate synchronic audio-visual events (Bahrick, 2004) that is available in ecologically relevant speech communication settings. The subjects are 5 to 6 month-old infants, who are assigned to different sessions. In one session, the subjects are presented with split-screen displaying four synchronized filmed variants of a speaker uttering different CV-syllables while the original sound track from one of the films was played. In another session, the subjects are presented with four alternative asynchronous faces of the same speaker along with a sound track that presents a non-speech sound synchronized with one of the alternatives. Looking times and detailed visual scanning of the speakers' faces are obtained using an eye-tracking system. No training or habituation to the audio-visual materials is provided. In yet another session, the infants are tested in their sensitivity to the 'McGurk effect' (McGurk & MacDonald, 1976). In this case the infants are previously familiarized with the words 'boll' and 'doll' that are heard while the corresponding objects are displayed. In a test phase immediately after the familiarization the subjects watch four films showing the speaker's face in between the two objects. The four film sequences are produced by combining each sound track with each of the video sequences. The looking times and the detailed eye-tracking are used to infer about the infant's integration of audio-visual speech information. Finally, a variant of this session was created by partially masking the audio signals (Fernald, Swingley, & Pinto, 2001) used during the test phase in order to assess the infants' 'phoneme restoration' ability in the presence of visual information. Research carried out within the frame of MILLE project, supported by The Bank of Sweden Tercentenary Foundation, by students from the Dpt of Clinical Science, Logopedics and Phoniatrics at Karolinska University Hospital, working at Dpt of Linguistics, Stockholm University, Sweden
Bibliographic reference. Lacerda, Francisco / Svensk, Anna / Molde, Charlotte / Olofsson, Jenny / Ristolainen, Linda / Carlson, Maria / Sj÷berg, Maria / Eklund, Marianne / Ekman-Brandt, Sara / Klintfors, Eeva (2005): "An ecological view on general mechanisms of early language acquisition", In PSP2005, 81 (abstract).