The topic of this workshop is "Multi-Modal Dialogues". I interpret this to mean dialogues which essentially Involve more than one mode, or medium, of communication. Most natural dialogues between people take this form. Speech may be the central mode, but it is generally augmented by visual Information, In particular, and this may be of various kinds with varying degrees of explicitness. Perhaps dialogues conducted by telephone are paradigm examples of pure speech interaction, and it is significant that most projections of the likely development of this technology include its Integration with a video channel, People want to be able to see each other when talking because much of the information transmitted during a natural dialogue Is transmitted visually.
My main purpose here, however, is to address Issues plausibly relevant to human-computer interaction (HCI), and this forces a narrowing of the focus to eliminate most of the non-verbal aspects of communication between people. It's Important, therefore, to distinguish the general field of non-verbal (visual) communication from that of graphical communication. The latter is intended to connote the explicit use of pictures as an essential part of a discourse, but more or less closely integrated with other media, such as NL (natural language, which may or may not involve speech, as drawings may illustrate a text in a book or a talk at a blackboard).
The use of such graphical media is thoroughly ubiquitous, I want to urge that NL dialogues which neglect this element are impoverished in practical terms, and restricted as examples for theoretical study. The added dimension of graphics Is critical for effective HCI, and the study of integrated dialogues should be central in the attempt to understand how human communication works, e.g. in cognitive terms. This discussion will survey a number of issues that emerge as important in studying Integrated NL/graphlcs dialogues.
Cite as: Lee, J. (1991) Graphics and natural language in multi-modal dialogues. Proc. 2nd VENACO Workshop - The Structure of Multimodal Dialogue, 165-168
@inproceedings{lee91_smmd,
author={John Lee},
title={{Graphics and natural language in multi-modal dialogues}},
year=1991,
booktitle={Proc. 2nd VENACO Workshop - The Structure of Multimodal Dialogue},
pages={165--168}
}