8th International Conference on Spoken Language Processing

Jeju Island, Korea
October 4-8, 2004

Integration Patterns during Multimodal Interaction

Anurag Kumar Gupta, Tasos Anastasakos

Motorola Inc, USA

The development of multimodal interfaces and algorithms for multimodal integration requires knowledge of integration patterns that represent how people use multiple modalities. We analyzed multimodal interaction with three different applications. Semantic analysis revealed that multimodal inputs can exhibit cooperation other than complementary and redundancy. Analysis of the relationship between tasks and multimodality revealed a positive correlation between the type of a task and multimodal interaction. In addition, analysis revealed that referential expressions can contain more information than simple deictic terms such as instructions to select or modify the referents.

Full Paper

Bibliographic reference.  Gupta, Anurag Kumar / Anastasakos, Tasos (2004): "Integration patterns during multimodal interaction", In INTERSPEECH-2004, 2293-2296.