FAAVSP - The 1st Joint Conference on Facial Analysis, Animation, and
Auditory-Visual Speech Processing

Vienna, Austria
September 11-13, 2015

Visual vs. Auditory Emotion Information: How language and Culture Affect Our Bias Towards the Different Modalities

Chee Seng Chong, Jeesun Kim, Chris Davis

MARCS Institute, University of Western Sydney, Australia

This study investigated if familiarity with a language that an emotion is expressed in, affects how information from the different sensory modalities are weighed in auditory-visual (AV) processing. The rationale for this study is that visual information may drive multisensory perception of emotion when a person is unfamiliar with a language, and this visual dominance effect may be reduced when a person is able to understand and extract emotion information from the language. To test this, Cantonese, English and Malay speakers were presented spoken Cantonese and English emotion expressions (angry, happy, sad, disgust and surprise) in AO, VO or AV conditions. Response matrices were examined to see if patterns of responses changed as a function of whether the expressions were produced in their native or non-native language. Our results show that the visual dominance effect for Cantonese and Malay participants changed depending on the language an emotion was expressed in, while the English participants showed a strong visual dominance effect regardless of the language of expression. Index Terms: vocal emotion perception, tone language, nontone language, AV perception, Cantonese, English, Malay

Full Paper

Bibliographic reference.  Chong, Chee Seng / Kim, Jeesun / Davis, Chris (2015): "Visual vs. auditory emotion information: how language and culture affect our bias towards the different modalities", In FAAVSP-2015, 46-51.