12th Annual Conference of the International Speech Communication Association

Florence, Italy
August 27-31. 2011

Synchronous Reading: Learning French Orthography by Audiovisual Training

Gérard Bailly, Will Barbour

GIPSA, France

We assess here the potential benefit of a karaoke-style reading system for learning sound-to-letter mapping in irregular languages. We have developed a framework that eases the development of interactive systems exploiting the alignment of text with audio at various levels (letters, phones syllables, words, chunks, etc). Synchronous reading consists of using time-aligned text with speech at the phone level to displace a cursor - here a virtual finger - on the text in synchrony with its verbalization. We demonstrate here that this bimodal reading implicitly facilitates the learning of the correspondence between sounds and letters in French for native and foreign subjects. Native subjects are shown to benefit more strongly from synchronous reading.

Full Paper

Bibliographic reference.  Bailly, Gérard / Barbour, Will (2011): "Synchronous reading: learning French orthography by audiovisual training", In INTERSPEECH-2011, 1153-1156.