11th Annual Conference of the International Speech Communication Association

Makuhari, Chiba, Japan
September 26-30. 2010

Prosody for the Eyes: Quantifying Visual Prosody Using Guided Principal Component Analysis

Erin Cvejic, Jeesun Kim, Chris Davis, Guillaume Gibert

University of Western Sydney, Australia

Although typically studied as an auditory phenomenon, prosody can also be conveyed by the visual speech signal, through increased movements of articulators during speech production, or through eyebrow and rigid head movements. This paper aimed to quantify such visual correlates of prosody. Specifically, the study was concerned with measuring the visual correlates of prosodic focus and prosodic phrasing. In the experiment, four participantsí speech and face movements were recorded while they completed a dialog exchange task with an interlocutor. Acoustic analysis showed that prosodic contrasts differed on duration, pitch and intensity parameters, which is consistent with previous findings in the literature. The visual data was processed using guided principal component analysis. The results showed that compared to the broad focused statement condition, speakers produced greater movement on both articulatory and non-articulatory parameters for prosodically focused and intonated words.

Full Paper

Bibliographic reference.  Cvejic, Erin / Kim, Jeesun / Davis, Chris / Gibert, Guillaume (2010): "Prosody for the eyes: quantifying visual prosody using guided principal component analysis", In INTERSPEECH-2010, 1433-1436.