ISCA Archive SSW 2021
ISCA Archive SSW 2021

Lipsyncing efforts for transcreating lecture videos in Indian languages

Mano Ranjith Kumar M, Jom Kuriakose, Karthik Pandia D S, Hema A Murthy

This paper proposes a novel lip-syncing module for the transcreation of lecture videos from English to Indian languages. The audio from the lecture is transcribed using automatic speech recognition. The text is translated and manually curated before and after translation to avoid mistakes. The curated text is synthesized using the Indian language end-to-endbased text-to-speech synthesis systems. The synthesized audio and video are out-of-sync. This paper attempts to automate this process of producing video lectures lip-synced into Indian languages using different techniques. Lip-syncing an educational video with the Indian language audio is challenging owing to (a) the duration of Indian language audio being considerably longer or shorter than that of the original audio, (b) the extempore speech causes the audio in the source videos to have long silences. Any modification to the speed of audio can be unpleasant to listeners. The proposed system non-uniformly re-samples the video to ensure better lip-syncing. The novelty of this paper is in the use of HMMGMM alignments in tandem with syllable segmentation using group delay, as visemes are closer to syllables. The proposed lip-syncing techniques are evaluated using subjective evaluation methods. Results indicate that accurate alignment at the syllable level is crucial for lip-syncing.


doi: 10.21437/SSW.2021-38

Cite as: M, M.R.K., Kuriakose, J., S, K.P.D., Murthy, H.A. (2021) Lipsyncing efforts for transcreating lecture videos in Indian languages. Proc. 11th ISCA Speech Synthesis Workshop (SSW 11), 216-221, doi: 10.21437/SSW.2021-38

@inproceedings{m21_ssw,
  author={Mano Ranjith Kumar M and Jom Kuriakose and Karthik Pandia D S and Hema A Murthy},
  title={{Lipsyncing efforts for transcreating lecture videos in Indian languages}},
  year=2021,
  booktitle={Proc. 11th ISCA Speech Synthesis Workshop (SSW 11)},
  pages={216--221},
  doi={10.21437/SSW.2021-38}
}