7th International Conference on Spoken Language Processing September 16-20, 2002 Denver, Colorado, USA

Multimodal Language Processing for Mobile Information Access

Michael Johnston, Srinivas Bangalore, Amanda Stent, Gunaranjan Vasireddy, Patrick Ehlen

AT&T Labs - Research, USA

Interfaces for mobile information access need to allow users flexibility in their choice of modes and interaction style in accordance with their preferences, the task at hand, and their physical and social environment. This paper describes the approach to multimodal language processing in MATCH (Multimodal Access To City Help), a mobile multimodal speech-pen interface to restaurant and subway information for New York City. Finite-state methods for multimodal integration and understanding enable users to interact using pen, speech, or dynamic combinations of the two, and a speech-act based multimodal dialogue manager enables mixed-initiative multimodal dialogue.

Full Paper

Bibliographic reference.  Johnston, Michael / Bangalore, Srinivas / Stent, Amanda / Vasireddy, Gunaranjan / Ehlen, Patrick (2002): "Multimodal language processing for mobile information access", In ICSLP-2002, 2237-2240.