7th International Conference on Spoken Language Processing

September 16-20, 2002
Denver, Colorado, USA

Neurocognitive Basis for Audiovisual Speech Perception: Evidence from Event-Related Potentials

Curtis W. Ponton (1), Edward T. Auer (2), Lynne E. Bernstein (2)

(1) Neuroscan Labs, USA; (2) House Ear Institute, USA

Knowledge about evoked potentials can be deployed to help explain audiovisual (AV) speech perception. The auditory and visual neural pathways are described in this paper, with emphasis on eventrelated potentials (ERPs) and their generation sites. The locations, interconnections, and neurophysiology of the auditory and visual pathways set limits on how and were information from auditory and visual speech signals combines. It is hypothesized that AV enhancement effects can occur at various levels of the cortical pathway, due to subcortical and to trans-cortical connections. Temporally early effects are likely localized to sub-cortical AV interactions that are not specialized for speech. Later temporal effects are more likely attributable to trans-cortical connections and feedback, including speech-specific processing. An experiment was conducted to examine the time course and cortical locations for AV speech interactions. ERPs were obtained during auditory-only (A), visual-only (V), and AV speech conditions. Dipole source modeling was used to localize and measure responses in auditory and visual cortical areas. At an early latency (100 ms), dipoles placed to measure superior temporal cortex activity showed enhancement (super-additivity relative to the sum of A and V responses) to AV speech; whereas, at a later latency (210 ms), the right occipital cortex dipole also showed super-additivity. Given the constraints of the cortical pathways, the early enhancement is interpreted as due to thalamo-cortical AV interactions, whereas, the later enhancement is interpreted as due to trans-cortical AV interactions.


Full Paper

Bibliographic reference.  Ponton, Curtis W. / Auer, Edward T. / Bernstein, Lynne E. (2002): "Neurocognitive basis for audiovisual speech perception: evidence from event-related potentials", In ICSLP-2002, 1697-1700.