ISCA Archive IberSPEECH 2022
ISCA Archive IberSPEECH 2022

Speaker-Adapted End-to-End Visual Speech Recognition for Continuous Spanish

David Gimeno-Gomez, Carlos David Martinez Hinarejos

Different studies have shown the importance of visual cues throughout the speech perception process. In fact, the development of audiovisual approaches has led to advances in the field of speech technologies. However, although noticeable results have recently been achieved, visual speech recognition remains an open research problem. It is a task in which, by dispensing with the auditory sense, challenges such as visual ambiguities and the complexity of modeling silence must be faced. Nonetheless, some of these challenges can be alleviated when the problem is approached from a speaker-dependent perspective. Thus, this paper studies, using the Spanish LIPRTVE database, how the estimation of specialized end-to-end systems for a specific person could affect the quality of speech recognition. First, different adaptation strategies based on the fine-tuning technique were proposed. Then, a pre-trained CTC/Attention architecture was used as a baseline throughout our experiments. Our findings showed that a two-step finetuning process, where the VSR system is first adapted to the task domain, provided significant improvements when the speaker adaptation was addressed. Furthermore, results comparable to the current state of the art were reached even when only a limited amount of data was available.


doi: 10.21437/IberSPEECH.2022-9

Cite as: Gimeno-Gomez, D., Hinarejos, C.D.M. (2022) Speaker-Adapted End-to-End Visual Speech Recognition for Continuous Spanish . Proc. IberSPEECH 2022, 41-45, doi: 10.21437/IberSPEECH.2022-9

@inproceedings{gimenogomez22_iberspeech,
  author={David Gimeno-Gomez and Carlos David Martinez Hinarejos},
  title={{Speaker-Adapted End-to-End Visual Speech Recognition for Continuous Spanish }},
  year=2022,
  booktitle={Proc. IberSPEECH 2022},
  pages={41--45},
  doi={10.21437/IberSPEECH.2022-9}
}