ISCA Archive Interspeech 2007
ISCA Archive Interspeech 2007

Complementarity and redundancy in multimodal user inputs with speech and pen gestures

Pui-Yu Hui, Zhengyu Zhou, Helen Meng

We present a comparative analysis of multi-modal user inputs with speech and pen gestures, together with their semantically equivalent uni-modal (speech only) counterparts. The multimodal interactions are derived from a corpus collected with a Pocket PC emulator in the context of navigation around Beijing. We devise a cross-modality integration methodology that interprets a multi-modal input and paraphrases it as a semantically equivalent, uni-modal input. Thus we generate parallel multimodal (MM) and unimodal (UM) corpora for comparative study. Empirical analysis based on class trigram perplexities shows two categories of data: (PPMM = PPUM) and (PPMM UM). The former involves complementarity across modalities in expressing the user's intent, including occurrences of ellipses. The latter involves redundancy, which will be useful for handling recognition errors by exploring mutual reinforcements. We present explanatory examples of data in these two categories.


doi: 10.21437/Interspeech.2007-600

Cite as: Hui, P.-Y., Zhou, Z., Meng, H. (2007) Complementarity and redundancy in multimodal user inputs with speech and pen gestures. Proc. Interspeech 2007, 2205-2208, doi: 10.21437/Interspeech.2007-600

@inproceedings{hui07_interspeech,
  author={Pui-Yu Hui and Zhengyu Zhou and Helen Meng},
  title={{Complementarity and redundancy in multimodal user inputs with speech and pen gestures}},
  year=2007,
  booktitle={Proc. Interspeech 2007},
  pages={2205--2208},
  doi={10.21437/Interspeech.2007-600}
}