ISCA Archive Interspeech 2008
ISCA Archive Interspeech 2008

Phone recognition from ultrasound and optical video sequences for a silent speech interface

Thomas Hueber, Gérard Chollet, Bruce Denby, Gérard Dreyfus, Maureen Stone

Latest results on continuous speech phone recognition from video observations of the tongue and lips are described in the context of an ultrasound-based silent speech interface. The study is based on a new 61-minute audiovisual database containing ultrasound sequences of the tongue as well as both frontal and lateral view of the speaker's lips. Phonetically balanced and exhibiting good diphone coverage, this database is designed both for recognition and corpus-based synthesis purposes. Acoustic waveforms are phonetically labeled, and visual sequences coded using PCA-based robust feature extraction techniques. Visual and acoustic observations of each phonetic class are modeled by continuous HMMs, allowing the performance of the visual phone recognizer to be compared to a traditional acoustic-based phone recognition experiment. The phone recognition confusion matrix is also discussed in detail.


doi: 10.21437/Interspeech.2008-528

Cite as: Hueber, T., Chollet, G., Denby, B., Dreyfus, G., Stone, M. (2008) Phone recognition from ultrasound and optical video sequences for a silent speech interface. Proc. Interspeech 2008, 2032-2035, doi: 10.21437/Interspeech.2008-528

@inproceedings{hueber08b_interspeech,
  author={Thomas Hueber and Gérard Chollet and Bruce Denby and Gérard Dreyfus and Maureen Stone},
  title={{Phone recognition from ultrasound and optical video sequences for a silent speech interface}},
  year=2008,
  booktitle={Proc. Interspeech 2008},
  pages={2032--2035},
  doi={10.21437/Interspeech.2008-528}
}