Error Handling in Spoken Dialogue Systems August 28-31, 2003 Château d'Oex, Vaud, Switzerland

Dialogues with Drawing

Atsushi Shimojima

Japan Advanced Institute of Science and Technology, ATR Media Information Science Labs, Kyoto, Japan

By "drawing", we mean the process of producing persistent traces on a surface. Thus, a prototypical example of drawing is the production of ink traces on a sheet of paper, but the notion also encompasses the production of chalk traces on a blackboard, that of pixel traces on a CRT screen, that of finger traces on sand, and so on. The main topic of this talk is not drawing itself, but the case where drawing is used with spoken language in interactive dialogues.

Dialogues with drawing are a very common form of human communication. People often talk while drawing: they give directions by drawing a map, explain ideas while writing on a blackboard, and explain a movement while drawing a curve. People often talk over a drawing: they discuss a travel plan over a road map, discuss a re-modeling over a blueprint, and discuss ideas referring to projected slides.

Partly because of this prevalence of dialogues involving drawing, they are envisioned as one of the promising forms of communication between humans and machines in the future, and explorations have begun on computational systems that help edit hand drawings, recognize them, or even interpret them in a limited domain. (See, for example, papers collected in the proceedings of the AAAI Spring Workshop on Sketch Understandings). A dialogue system that interprets and produces natural drawing is therefore not an unrealistic idea.

Dialogues involving drawing are also an interesting subject of study. Speech is a non-persistent medium of communication, in that spoken sound is available to auditory perception only while it is produced, and it generally leaves no perceivable trace after production. Drawing is, by definition, a persistent medium of communication. Also, drawing generally has a broader bandwidth than speech does: two or more speakers can produce drawing simultaneous on a sufficiently large drawing surface, while spoken sounds produced simultaneously by two or more speakers often interfere with each other and become incomprehensible. It is plausible that dialogues conducted mainly with speech are so structured as to handle the non-persistency and narrow bandwidth of the speech medium, so the presence of drawing can make significant differences in the structure of dialogues. Detailed comparisons can reveal the functions of important structural features for both types of dialogues.

Furthermore, what is drawn in communication is often graphics, such as maps, floor plans, and line figures, as distinguished from words, phrases, and sentences in a certain language. Detailed analyses of the use of drawing in real dialogues can therefore lead to important findings about the expressive functions of graphical representations---the subject vigorously studied in a certain community of cognitive scientists (e.g., Larkin and Simon 1987, Barwise and Etchemendy 1990, Stenning and Oberlander 1995, Shimojima 1999).

The purpose of this talk is to introduce the audience to the subject of dialogues with drawing. I will give an overview of the key studies that have been conducted on the subject, with some focus on our recent empirical work (Umata, Shimojima, and Katagiri, 1999, 2000, 2001, 2002, Takeoka, Shimo jima, and Katagiri 2003). In particular, I will show some relevant data concerning the following questions:

• Does drawing constitute a turn, as speech typically does? Does drawing by one speaker resist interruptions by the other speaker's speech? Does the occurence of drawing give an excuse to long silence at the speech level?
• Under what conditions is a "crush of drawing turns" (simultaneous drawing on the shared drawing surface) permissible? Under what conditions does drawing have to occur in a sequential order as speech turns typically do?
• How are speech and drawing integrated in conveying information in dialogues? Does the existence of a drawing make any differences in the use of spoken language?

I will also try to give an overview of the areas not covered by the previous research, and clarify some of the important research questions in those areas.

Bibliographic reference.  Shimojima, Atsushi (2003): "Dialogues with drawing", In EHSD-2003, 23 (Abstract).