This paper adresses two questions. The first question is if perception of emotional prosody changes when visual stimuli are present. It has already been shown by others (e.g. Massaro) that the perception of vowels and consonants is augmented or changed when visual information is present.The question is then if perception of such categories as emotions, mainly expressed by prosody, are also affected by multimodal stimuli. The second question concerns the cross-linguistic perception of multimodal stimuli. Earlier crosscultural, studies indicate that facial expression of emotions is more universal than prosody is. Crosslinguistic interpretation of emotions could then be more successful multimodally than only vocally. The specific questions asked in the present study are to what degree Spanish and Swedish listeners can interpret Spanish emotional prosody, and whether simultaneously presented faces, expressing the same emotions, change or improve the interpretation. Audio recordings of Spanish emotional expressions were presented to Spanish and Swedish listeners, in two experimental settings. In the first setting the listeners only attended to prosody, in the second one they also saw a face, expressing different emotions. The results indicate that intra-linguistic as well as cross-linguistic perception of emotional prosody is improved by visual stimuli. Cross-linguistic interpretation of prosody is more poorly accomplished than interlinguistic, but seems to be greatly augmented by multimodal stimuli. Acoustic analysis of the stimuli are made and discussed in relation to the results of he perceptions of the stimuli of the different language groups. The results are also discussed in terms of age and gender of listeners.
Cite as: Abelin, Å. (2005) Change of perception of emotions in multimodal and crosslinguistic settings. Proc. ISCA Workshop on Plasticity in Speech Perception (PSP 2005), 113-116
@inproceedings{abelin05_psp,
author={Åsa Abelin},
title={{Change of perception of emotions in multimodal and crosslinguistic settings}},
year=2005,
booktitle={Proc. ISCA Workshop on Plasticity in Speech Perception (PSP 2005)},
pages={113--116}
}