Speech and Language Technology in Education (SLaTE2007)

The Summit Inn, Farmington, PA, USA
October 1-3, 2007

Using Visual Speech for Training Chinese Pronunciation: An In-vivo Experiment

Ying Liu (1), Dominic W. Massaro (2), Trevor H. Chen (2), Derek Chan (1), Charles Perfetti (1)

(1) Learning Research and Development Center, University of Pittsburgh, PA, USA
(2) Department of Psychology, Perceptual Science Laboratory, University of California, Santa Cruz, CA, USA

Recent research showed that our perception and understanding of speech are influenced by a speaker's facial expressions and accompanying gestures, as well as the actual sound of the speech. Perceivers expertly use these multiple sources of information to identify and interpret the language input. Baldi® is a three-dimensional animated talking head appropriately aligned with either synthesized or natural speech. The present in-vivo experiment used Bao, a Chinese version of Baldi, to teach Chinese syllables to adult English native speakers. The result showed that students trained with Baldi, improved more than students trained with ordinary speech. Advantages of the Baldi pedagogy and technology include the popularity and proven effectiveness of computers and embodied conversational agents, the perpetual availability of the program, and individualized instruction. The technological edge of Baldi holds great promise in language learning, dialog, human-machine interaction, education, and edutainment.

Full Paper

Bibliographic reference.  Liu, Ying / Massaro, Dominic W. / Chen, Trevor H. / Chan, Derek / Perfetti, Charles (2007): "Using visual speech for training Chinese pronunciation: an in-vivo experiment", In SLaTE-2007, 29-32.