5th International Conference on Spoken Language Processing

Sydney, Australia
November 30 - December 4, 1998

A Speechreading Aid Based on Phonetic ASR

Paul Duchnowski, Louis Braida, Maroula Bratakos, David Lum, Matthew Sexton, Jean Krause

Massachusetts Institute of Technology, USA

Manual Cued Speech (MCS) is an effective method of communication by the deaf and hearing-impaired. We describe our work on assessing the feasibility of automatic determination and presentation of cues without intervention by the speaker. The conclusions of this study are applied to the design and implementation of a prototype automatic cueing system using HMM-based automatic speech recognition software to identify the cues in real time. We also describe the features of our cue display that enhance its effectiveness such as style of cue images and the timing of their transitions. Our experiments show keyword reception by experienced MCS users improves significantly with the use of our system (66%) relative to speechreading alone (35%) on low-context sentences.

Full Paper
Video #1 [.mpg] - The manually cued sentence "The old castle passed from the duke to the king."
Video #2 [.mpg] - Automatically cued (discrete cues) sentence "The loss and two wins were fair games."
Video #3 [.mpg] - Automatically cued (dynamic cues) sentence "The kite may fly on this windy day."

Bibliographic reference.  Duchnowski, Paul / Braida, Louis / Bratakos, Maroula / Lum, David / Sexton, Matthew / Krause, Jean (1998): "A speechreading aid based on phonetic ASR", In ICSLP-1998, paper 0589.