INTERSPEECH 2011 12th Annual Conference of the International Speech Communication Association Florence, Italy August 27-31. 2011

Crossmodal Prosodic and Gestural Contribution to the Perception of Contrastive Focus

Pilar Prieto (1), Cecilia Pugliesi (2), Joan Borràs-Comes (2), Ernesto Arroyo (2), Josep Blat (2)

(1) ICREA, Spain
(2) Universitat Pompeu Fabra, Spain

Speech prosody has traditionally been analyzed in terms of acoustic features. Even though visual features and gestures have been shown to help and enhance linguistic processing, the conventional view is that facial and body gesture information in oral (non-sign) languages tends to be redundant and has the role of helping the hearer recover the meaning of an utterance. We conducted two perception experiments with a 3D animated character showing conflicting auditory and visual information to investigate two related questions regarding the importance of gestures in conveying prosodic meaning: (a) how important are facial cues with respect to auditory cues for the perception of contrastive focus?; and (b) what is the relevance of the different gestural movements (i.e., head nod and eyebrow raising) for the perception of this type of focus? Our findings reveal that the visual component is crucial in the semantic interpretation of contrastive focus.

Full Paper

Bibliographic reference.  Prieto, Pilar / Pugliesi, Cecilia / Borràs-Comes, Joan / Arroyo, Ernesto / Blat, Josep (2011): "Crossmodal prosodic and gestural contribution to the perception of contrastive focus", In INTERSPEECH-2011, 977-980.