Auditory-Visual Speech Processing (AVSP) 2010

Hakone, Kanagawa, Japan
September 30-October 3, 2010

CENSREC-1-AV: An Audio-Visual Corpus for Noisy Bimodal Speech Recognition

Satoshi Tamura (1), Chiyomi Miyajima (2), Norihide Kitaoka (2), Takeshi Yamada (3), Satoru Tsuge (4), Tetsuya Takiguchi (5), Kazumasa Yamamoto (6), Takanobu Nishiura (7), Masato Nakayama (8), Yuki Denda (9), Masakiyo Fujimoto (10), Shigeki Matsuda (11), Tetsuji Ogawa (12), Shingo Kuroiwa (13), Kazuya Takeda (2), Satoshi Nakamura (11)

(1) Gifu University; (2) Nagoya University; (3) University of Tsukuba; (4) Daido University; (5) Kobe University; (6) Toyohashi University of Technology; (7) Ritsumeikan University; (8) Kinki University; (9) Murata Machinery; (10) NTT CS Laboratory; (11) NiCT/ATR; (12) Waseda University; (13) Chiba University; Japan

In this paper, an audio-visual speech corpus CENSREC-1-AV for noisy speech recognition is introduced. CENSREC-1-AV consists of an audio-visual database and a baseline system of bimodal speech recognition which uses audio and visual information. In the database, there are 3,234 and 1,963 utterances made by 42 and 51 speakers as a training and a test sets respectively. Each utterance consists of a speech signal as well as color and infrared pictures around a speaker’s mouth. A baseline system is built so that a user can evaluate a proposed bimodal speech recognizer. In the baseline system, multi-stream HMMs are obtained using training data. A preliminary experiment was conducted to evaluate the baseline using acoustically noisy testing data. The results show that roughly a 35% relative error reduction was achieved in low SNR conditions compared with an audio-only ASR method.

Index Terms: audio-visual database, bimodal speech recognition, noise robustness, eigenface, optical flow.

Full Paper

Bibliographic reference.  Tamura, Satoshi / Miyajima, Chiyomi / Kitaoka, Norihide / Yamada, Takeshi / Tsuge, Satoru / Takiguchi, Tetsuya / Yamamoto, Kazumasa / Nishiura, Takanobu / Nakayama, Masato / Denda, Yuki / Fujimoto, Masakiyo / Matsuda, Shigeki / Ogawa, Tetsuji / Kuroiwa, Shingo / Takeda, Kazuya / Nakamura, Satoshi (2010): "CENSREC-1-AV: an audio-visual corpus for noisy bimodal speech recognition", In AVSP-2010, paper P6.