5th International Conference on Spoken Language Processing

Sydney, Australia
November 30 - December 4, 1998

Referential Features and Linguistic Indirection in Multimodal Language

Sharon L. Oviatt (1), Karen Kuhn (2)

(1) Oregon Graduate Institute, USA
(2) AlTech, USA

The present report outlines differences between multimodal and unimodal communication patterns in linguistic features associated with ease of dialogue tracking and ambiguity resolution. A simulation method was used to collect data while participants used spoken, pen-based, or multimodal input during spatial tasks with a dynamic system. Users' linguistic constructions were analyzed for differences in the rates of reference, co-reference, definite and indefinite referring expressions, and deictic terms. Differences also were summarized in the prevalence of linguistic indirection. Results indicate that spoken language contains substantially higher levels of referring and co-referring expressions and also linguistic indirection, compared with multimodal language communicated by the same users completing the same task. In contrast, multimodal language not only has fewer referential expressions and relatively little anaphora, it also specifically lacks the regular use of determiners observed in spoken definite and indefinite noun phrases. In addition, multimodal language is distinct in its high levels of deictic reference. Implications of these findings are discussed for the relative ease of natural language processing for speech-only versus multimodal systems.

Full Paper

Bibliographic reference.  Oviatt, Sharon L. / Kuhn, Karen (1998): "Referential features and linguistic indirection in multimodal language", In ICSLP-1998, paper 0048.