Multimodal interfaces enable more natural and effective human-computer interaction by providing multiple channels through which input or output may pass. In order to realize their full potential, they need to support not just input from multiple modes, but synchronized integration of modes. This paper describes a multimodal language processing architecture which allows for declarative statement of multimodal integration strategies in a unification-based grammar formalism. The architecture is currently deployed in a working system supporting interaction with dynamic maps using speech and pen, but the approach is more general and extends to a wide variety of other potential multimodal interfaces.
Cite as: Johnston, M. (1998) Multimodal language processing. Proc. 5th International Conference on Spoken Language Processing (ICSLP 1998), paper 0893, doi: 10.21437/ICSLP.1998-278
@inproceedings{johnston98_icslp,
author={Michael Johnston},
title={{Multimodal language processing}},
year=1998,
booktitle={Proc. 5th International Conference on Spoken Language Processing (ICSLP 1998)},
pages={paper 0893},
doi={10.21437/ICSLP.1998-278}
}