Auditory-Visual Speech Processing (AVSP'98)

December 4-6, 1998
Terrigal - Sydney, Australia

Integration of Audiovisually Compatible and Incompatible Consonants in Identification Experiments

Louis D. Braida, Kaoru Sekiyama, Ann K. Dix

Research Laboratory of Electronics, Massachusetts Institute of Technology, USA

A mathematical model that has successfully accounted for audiovisual integration of natural consonants with compatible audio and visual components is applied to artificial stimuli with incompatible components. Twenty eight Japanese subjects identified stimuli derived from the eight consonants /b,d,g,p,t,k,m,n/ under auditory, visual, and audiovisual conditions. The acoustic stimulus was degraded by lowpass filtering and additive noise. A single set of model parameters, derived from confusion matrices obtained in the unimodal experiments, was found to make fairly good predictions of the results obtained in the audiovisual experiment for both the compatible and incompatible stimuli. Limitations of the approach and implications of the results are discussed.


Full Paper

Bibliographic reference.  Braida, Louis D. / Sekiyama, Kaoru / Dix, Ann K. (1998): "Integration of audiovisually compatible and incompatible consonants in identification experiments", In AVSP-1998, 49-54.