Auditory-Visual Speech Processing (AVSP'98)

December 4-6, 1998
Terrigal - Sydney, Australia

Is it Possible to Evaluate the Contribution of Visual Information to the Process of Speech Comprehension?

Loredana Cerrato, Federico Albano Leoni, Mauro Falcone

Fondazione Ugo Bordoni, (Italy)

We report the results of a series of comprehension tests run with the aim of investigating the contribution of visual information to the process of comprehension of conversational speech. Using as sample material a short conversation held by two male speakers, edited from an Italian TV soap opera, we run 3 comprehension tests: 1. submission of the multi-modal speech signal (auditory + visual); 2. submission of the sample only in the auditory modality (i.e. without the integration of visual cues); 3. submission of the sample only in the visual modality (without the integration of auditory cues). It is clear from our results that the visual cues help the subjects to understand the main topic of conversation and to remember some of the details of the conversation. Moreover they seem to play an important role for the interpretation of the emotional state of the speakers. In some cases visual cues appear to be misleading.


Full Paper

Bibliographic reference.  Cerrato, Loredana / Leoni, Federico Albano / Falcone, Mauro (1998): "Is it possible to evaluate the contribution of visual information to the process of speech comprehension?", In AVSP-1998, 141-146.