Third European Conference on Speech Communication and Technology

Berlin, Germany
September 22-25, 1993


A New Interface Paradigm: Automatic Recognition of Integrated Speech and Handwriting Information

Jerome R. Bellegarda, Dimitri Kanevsky

IBM Research Division, TJ. Watson Research Center, Yorktown Heights, NY, USA

Because of the difficulties inherent to the analysis of such complex processes as speaking and (hand)writing, the machine recognition of the two most natural ways of communicating has so far met with only limited success. In this paper we consider a new paradigm: the integrated use of speech and handwriting information to improve the overall accuracy of an automatic recognizer. This approach is made possible by the complementarity of the two sources of information yielding the acoustic and pen stroke evidence. It leads to the development of ISWI, the Integrated Speech and Writing Interface. Preliminary results indicate the viability of ISWI, particularly in applications such as human factor studies. These results also bring useful insights into the feasibility of ISWI given the present level of speech and handwriting recognition performance. It seems reasonable to expect ISWI to become the logical choice for the ultimate user-friendly man-machine interface.

Full Paper

Bibliographic reference.  Bellegarda, Jerome R. / Kanevsky, Dimitri (1993): "A new interface paradigm: automatic recognition of integrated speech and handwriting information", In EUROSPEECH'93, 2157-2160.