5th International Conference on Spoken Language Processing

Sydney, Australia
November 30 - December 4, 1998

Multimodal Language Processing

Michael Johnston

Oregon Graduate Institute, USA

Multimodal interfaces enable more natural and effective human-computer interaction by providing multiple channels through which input or output may pass. In order to realize their full potential, they need to support not just input from multiple modes, but synchronized integration of modes. This paper describes a multimodal language processing architecture which allows for declarative statement of multimodal integration strategies in a unification-based grammar formalism. The architecture is currently deployed in a working system supporting interaction with dynamic maps using speech and pen, but the approach is more general and extends to a wide variety of other potential multimodal interfaces.

Full Paper

Bibliographic reference.  Johnston, Michael (1998): "Multimodal language processing", In ICSLP-1998, paper 0893.