Auditory-Visual Speech Processing (AVSP) 2011

Volterra, Italy
September 1-2, 2011

Turn-Taking Control Using Gaze in Multiparty Human-Computer Dialogue: Effects of 2D and 3D Displays

Samer Al Moubayed, Gabriel Skantze

Department of Speech, Music and Hearing, KTH Royal Institute of Technology, Stockholm, Sweden

In a previous experiment we found that the perception of gaze from an animated agent on a two-dimensional display suffers from the Mona Lisa effect, which means that exclusive mutual gaze cannot be established if there is more than one observer. By using a three-dimensional projection surface, this effect can be eliminated. In this study, we investigate whether this difference also holds for the turn-taking behaviour of subjects interacting with the animated agent in a multi-party dialogue. We present a Wizard-of-Oz experiment where five subjects talk to an animated agent in a route direction dialogue. The results show that the subjects to some extent can infer the intended target of the agent’s questions, in spite of the Mona Lisa effect, but that the accuracy of gaze when it comes to selecting an addressee is still significantly lower in the 2D condition, as compared to the 3D condition. The response time is also significantly longer in the 2D condition, indicating that the inference of intended gaze may require additional cognitive efforts.

Index Terms. Turn-taking, Multi-party Dialogue, Gaze, Facial Interaction, Mona Lisa Effect, Facial Projection, Wizard of Oz

Full Paper

Bibliographic reference.  Al Moubayed, Samer / Skantze, Gabriel (2011): "Turn-taking control using gaze in multiparty human-computer dialogue: effects of 2d and 3d displays", In AVSP-2011, 99-102.