SPOKEN WORD ACCESS PROCESSES (SWAP) May 29-31, 2000 Nijmegen, The Netherlands

Feedback in audiovisual speech perception

Carol A. Fowler (1), Lawrence Brancazio (2)

(1) Haskins Laboratories
(2) Northeastern University

Lexical information affects identification of ambiguous phones (Ganong, 1980). Contrary to interactive models of auditory word recognition, recent evidence suggests that this effect is not due to feedback from a lexical to a phonemic processing layer. A phone ambiguous between /s/ and /S/ identified as /s/ following /dZu/ (making "juice" rather than "joosh") does not engender compensation for coarticulatiory effects of /s/ on a following /t/ or /k/ (Pitt & McQueen, 1998). However, compensation for coarticulation by phones identified bottom-up does occur. Because compensation is assumed to be prelexical, these results imply distinct phonological and lexical processing stages linked by a feedforward-only relation.

Lexical information also affects phoneme identification when optical and acoustic information for speech gestures are discrepant (Brancazio, 1998): optical /d/ dubbed onto acoustic /b/ leads to fewer /b/ percepts preceding audiovisual /Ens/ (bense-dense) than preceding /EntS/ (bench-dench). Audio-visual integration also fosters compensation for coarticulation; a visual /l/ dubbed onto an ambiguous acoustic signal increases identification of a following /g/ (Fowler, Brown & Mann, in press). This suggests that audiovisual integration is prelexical. 

Our question now is where the lexical effect on audiovisual integration occurs. Does it occur at a lexical stage of processing after audiovisual integration produces bottom-up ambiguity? Or is the effect due to feedback from a lexical level of processing to one at which audiovisual integration and compensation for coarticulation occur?

We are currently running experiments to address this issue. We are testing whether an audiovisually-induced compensation for coarticulation effect will be modulated by a lexical influence on audiovisual integration. A finding that the magnitude of the compensation effect shifts with lexically-induced variation in the visual influence in audiovisual integration would imply that lexical information feeds back onto audiovisual integration. A lexical effect on audiovisual integration that had no effect on compensation for coarticulation would be consistent with a feedforward-only account.

References