Silent Speech Interfaces use data from the speech production process, such as visual information of face movements. However, using a single modality limits the amount of available information. In this study we start to explore the use of multiple data input modalities in order to acquire a more complete representation of the speech production model. We have selected 4 non-invasive modalities – Visual data from Video and Depth, Surface Electromyography and Ultrasonic Doppler - and created a system that explores the synchronous combination of all 4, or of a subset of them, into a multimodal Silent Speech Interface (SSI). This paper describes the system design, data collection and first word recognition results. As the first acquired corpora are necessarily small for this SSI, we use for classification an example based recognition approach based on Dynamic Time Warping followed by a weighted k-Nearest Neighbor classifier. The first classification results using different vocabularies, with digits, a small set of commands related to Ambient Assisted Living and minimal nasal pairs, show that word recognition benefits can be obtained from a multimodal approach.
Index Terms: silent speech interfaces, multimodal, video and depth information
Cite as: Freitas, J., Teixeira, A., Dias, M.S. (2013) Multimodal silent speech interface based on video, depth, surface electromyography and ultrasonic Doppler: data collection and first recognition results. Proc. Speech Production in Automatic Speech Recognition (SPASR-2013), 44-49
@inproceedings{freitas13_spasr,
author={João Freitas and António Teixeira and Miguel Sales Dias},
title={{Multimodal silent speech interface based on video, depth, surface electromyography and ultrasonic Doppler: data collection and first recognition results}},
year=2013,
booktitle={Proc. Speech Production in Automatic Speech Recognition (SPASR-2013)},
pages={44--49}
}