Automatic Detection of the Temporal Segmentation of Hand Movements in British English Cued Speech

Li Liu, Jianze Li, Gang Feng, Xiao-Ping Zhang


Cued Speech (CS) is a multi-modal system, which complements the lip reading with manual hand cues in the phonetic level to make the spoken language visible. It has been found that lip and hand movements are asynchronous in CS, and thus the study of hand temporal organization is very important for the multi-modal CS feature fusion. In this work, we propose a novel diphthong-hand preceding model (D-HPM) by investigating the relationship between hand preceding time (HPT) and diphthong time instants in sentences for British English CS. Besides, we demonstrate that HPT of the first and second parts of diphthongs has a very strong correlation. Combining the monophthong-HPM (M-HPM) and D-HPM, we present a hybrid temporal segmentation detection algorithm (HTSDA) for the hand movement in CS. The evaluation of the proposed algorithm is carried out by a hand position recognition experiment using the multi-Gaussian classifier as well as the long-short term memory (LSTM). The results show that the HTSDA significantly improves the recognition performance compared with the baseline (i.e., audio-based segmentation) and the state-of-the-art M-HPM. To the best of our knowledge, this is the first work to study the temporal organization of hand movements in British English CS.


 DOI: 10.21437/Interspeech.2019-2353

Cite as: Liu, L., Li, J., Feng, G., Zhang, X. (2019) Automatic Detection of the Temporal Segmentation of Hand Movements in British English Cued Speech. Proc. Interspeech 2019, 2285-2289, DOI: 10.21437/Interspeech.2019-2353.


@inproceedings{Liu2019,
  author={Li Liu and Jianze Li and Gang Feng and Xiao-Ping Zhang},
  title={{Automatic Detection of the Temporal Segmentation of Hand Movements in British English Cued Speech}},
  year=2019,
  booktitle={Proc. Interspeech 2019},
  pages={2285--2289},
  doi={10.21437/Interspeech.2019-2353},
  url={http://dx.doi.org/10.21437/Interspeech.2019-2353}
}