This paper describes the multimodal system, which was developed by two laboratories of SPIIRAS, for assistance to people with disabilities of hands. It combines automatic speech recognition and head tracking in joint multimodal system. The structure of the system, used methods for recognition and tracking, information fusion and synchronization, obtained results and testing conditions are described in the paper. Finally, the developed system was applied for hands-free operation for Graphical User Interface in such common tasks as communication by Internet and text edition by MS Word. The experiments have shown that in spite of some decreasing of operation speed the multimodal system allows to work with computer without using standard mouse and keyboard. Thus the developed assistive multimodal system can be successfully used for hands-free PC control for users with disabilities of their hands or arms.
Cite as: Karpov, A.A., Ronzhin, A.L., Nechaev, A.I., Chernakova, S.E. (2004) Assistive multimodal system based on speech recognition and head tracking. Proc. 9th Conference on Speech and Computer (SPECOM 2004), 521-530
@inproceedings{karpov04b_specom,
author={Alexey A. Karpov and Andrey L. Ronzhin and Alexander I. Nechaev and Svetlana E. Chernakova},
title={{Assistive multimodal system based on speech recognition and head tracking}},
year=2004,
booktitle={Proc. 9th Conference on Speech and Computer (SPECOM 2004)},
pages={521--530}
}