MobiVSR : Efficient and Light-Weight Neural Network for Visual Speech Recognition on Mobile Devices

Nilay Shrivastava, Astitwa Saxena, Yaman Kumar, Rajiv Ratn Shah, Amanda Stent, Debanjan Mahata, Preeti Kaur, Roger Zimmermann


Visual speech recognition (VSR) is the task of recognizing spoken language from video input only, without any audio. VSR has many applications as an assistive technology, especially if it could be deployed in mobile devices and embedded systems. The need for intensive computational resources and large memory footprint are two major obstacles in deploying neural network models for VSR in a resource constrained environment. We propose a novel end-to-end deep neural network architecture for word level VSR called MobiVSR with a design parameter that aids in balancing the model’s accuracy and parameter count. We use depthwise 3D convolution along with channel shuffling for the first time in the domain of VSR and show how it makes our model efficient. MobiVSR achieves an accuracy of 70% on a challenging Lip Reading in the Wild dataset with 6 times fewer parameters and 20 times smaller memory footprint than the current state of the art. MobiVSR can also be compressed to 6 MB by applying post training quantization.


 DOI: 10.21437/Interspeech.2019-3273

Cite as: Shrivastava, N., Saxena, A., Kumar, Y., Shah, R.R., Stent, A., Mahata, D., Kaur, P., Zimmermann, R. (2019) MobiVSR : Efficient and Light-Weight Neural Network for Visual Speech Recognition on Mobile Devices. Proc. Interspeech 2019, 2753-2757, DOI: 10.21437/Interspeech.2019-3273.


@inproceedings{Shrivastava2019,
  author={Nilay Shrivastava and Astitwa Saxena and Yaman Kumar and Rajiv Ratn Shah and Amanda Stent and Debanjan Mahata and Preeti Kaur and Roger Zimmermann},
  title={{MobiVSR : Efficient and Light-Weight Neural Network for Visual Speech Recognition on Mobile Devices}},
  year=2019,
  booktitle={Proc. Interspeech 2019},
  pages={2753--2757},
  doi={10.21437/Interspeech.2019-3273},
  url={http://dx.doi.org/10.21437/Interspeech.2019-3273}
}