Sixth International Conference on Spoken Language Processing
(ICSLP 2000)

Beijing, China
October 16-20, 2000

Multimodal Signal Processing in Naturalistic Noisy Environments

Sharon Oviatt

Department of Computer Science and Engineering, Oregon Graduate Institute, Beaverton, OR, USA

When a system must process spoken language in natural environments that involve different types and levels of noise, the problem of supporting robust recognition is a very difficult one. In the present studies, over 2,600 multimodal utterances were collected during both mobile and stationary use of a multimodal pen/voice system. The results confirmed that multimodal signal processing supports significantly improved robustness over spoken language processing alone, with the largest improvement during mobile use. The multimodal architecture decreased the spoken language error rate by 19-35%. In addition, data collected on a command-by-command basis while users were mobile emphasized the adverse impact of users’ Lombard adaptation on system processing, even when a noise-canceling microphone was used. Implications of these findings are discussed for improving the reliability and stability of spoken language processing in mobile environments.


Full Paper

Bibliographic reference.  Oviatt, Sharon (2000): "Multimodal signal processing in naturalistic noisy environments", In ICSLP-2000, vol.2, 696-699.