7th International Conference on Spoken Language Processing

September 16-20, 2002
Denver, Colorado, USA

Gestural Spatialization in Natural Discourse Segmentation

Francis Quek (1), David McNeill (2), Robert Bryll (1), Mary Harper (3)

(1) Wright State University, USA; (2) University of Chicago, USA; (3) Purdue University, USA

Human multimodal communicative behaviors form a tightly integrated whole. By matching up gestural features with a carefully time-tagged transcription of the speech, we can observe how gesture features and discourse unit transitions cohere. Space usage SU is a key gestural component. We summarize the theory of SU. In our experiments where subjects make action plans around a terrain map, such SU become key organizational loci around which the discourse may be built. Our vision-based approach extracts ‘SU histograms’ from stereo video describing the locus of motion of a speaker’s dominant hand. An N ×N ‘fuzzy correlation’ of these histograms yields a correlation space in which similar SU is clustered. By locating the cluster transitions we can locate topical shifts in the discourse. We show results by comparing the transitions extracted from a sentential coding with a psycholinguistic semantic coding. We do the same with a uniform distributed time units and demonstrate the ability to recover discourse transitions.

Full Paper

Bibliographic reference.  Quek, Francis / McNeill, David / Bryll, Robert / Harper, Mary (2002): "Gestural spatialization in natural discourse segmentation", In ICSLP-2002, 189-192.