InSTIL/ICALL 2004 Symposium on Computer Assisted Learning June 17-19, 2004 Venice, Italy

Analyzing the Interplay Between Spoken Language and Gestural Cues in Conversational Child-Machine Interactions in Pre/Early Literate Age Groups

Simona Montanari, Serdar Yildirim, Sonia Khurana, Marni Landes, Lewis Lawyer, Elaine Andersen, Shrikanth Narayanan

Integrated Media Systems Center, Speech Analysis and Interpretation Lab, Department of Linguistics, Department of Electrical Engineering, University of Southern California, Los Angeles, CA, USA

This paper reports on analysis of interplay between verbal and gestural cues in conversational childmachine interactions in pre-literate age groups. In particular, this study focuses on children’s responses to trouble spots (intra-utterance self-repairs, filled pauses), reference marking, and gestural cues (i.e. pointing or touching screen) that children employ in computerdirected children-computer interaction using a data from 15 children, 3 to 6 years old. Our gestural analysis indicated that with increasing age, children rely on purely verbal utterances rather than utterances accompanied by gestures. Co-analysis of gestural use and intra-utterance repairs revealed that children were less likely to produce intra-utterance repairs when they are multimodal. Also, results revealed that children were less likely to use filled pauses when they are interacting multimodally. In addition, the more children rely on purely verbal utterance, the more they use fullform, explicit noun phrases rather than pro-forms such as deictic adverbs and deictic numerals.

Full Paper

Bibliographic reference.  Montanari, Simona / Yildirim, Serdar / Khurana, Sonia / Landes, Marni / Lawyer, Lewis / Andersen, Elaine / Narayanan, Shrikanth (2004): "Analyzing the interplay between spoken language and gestural cues in conversational child-machine interactions in pre/early literate age groups", In ICALL-2004, paper 047.