Generation of Synthetic Sign Language Sentences

Aitana Villaplana, Carlos David Martinez Hinarejos

Sign language is one of the most usual ways of communication for deaf people. Their inclusion in the society would be greatly improved if sign language can be easily used to communicate with other people that do not understand properly that language. Automatic recognition systems, based on machine learning techniques, could be very useful for this task, providing signers with tools that could be used to transcribe sign language into written language automatically. Many previous works have centered mainly in the recognition of single words, and different datasets of single words signs are available for estimating recognition models for this task. However, the recognition of whole sentences is difficult, since the acquisition of datasets of sentences is in general harder than the acquisition of single words. Thus, the possibility of generating sentences in sign language from single word datasets is very attractive to obtain automatic systems for decoding sign language sentences. In this work, we present an approximation for generating sign sentences from sign single words acquired by using the LeapMotion sensor. We study the different difficulties that present this generation process. Results for real sign language sentences show that training with these synthetic sentences improves the decoding performance with respect to using only single words for training.

doi: 10.21437/IberSPEECH.2021-50

Villaplana, A, Martinez Hinarejos, C.D (2021) Generation of Synthetic Sign Language Sentences. Proc. IberSPEECH 2021, 235-239, doi: 10.21437/IberSPEECH.2021-50.