SPECOM'2004: 9th Conference
Speech and Computer

Saint-Petersburg, Russia
September 20-22, 2004

Assistive Multimodal System Based on Speech Recognition and Head Tracking

Alexey A. Karpov, Andrey L. Ronzhin (1), Alexander I. Nechaev, Svetlana E. Chernakova (2)

(1) Speech Informatics Group; (2) Robotics Laboratory; St. Petersburg Institute for Informatics and Automation of the Russian Academy of Sciences, Russia

This paper describes the multimodal system, which was developed by two laboratories of SPIIRAS, for assistance to people with disabilities of hands. It combines automatic speech recognition and head tracking in joint multimodal system. The structure of the system, used methods for recognition and tracking, information fusion and synchronization, obtained results and testing conditions are described in the paper. Finally, the developed system was applied for hands-free operation for Graphical User Interface in such common tasks as communication by Internet and text edition by MS Word. The experiments have shown that in spite of some decreasing of operation speed the multimodal system allows to work with computer without using standard mouse and keyboard. Thus the developed assistive multimodal system can be successfully used for hands-free PC control for users with disabilities of their hands or arms.

Full Paper

Bibliographic reference.  Karpov, Alexey A. / Ronzhin, Andrey L. / Nechaev, Alexander I. / Chernakova, Svetlana E. (2004): "Assistive multimodal system based on speech recognition and head tracking", In SPECOM-2004, 521-530.