ISCA Archive Interspeech 2008
ISCA Archive Interspeech 2008

Fusion of audio and video modalities for detection of acoustic events

Taras Butko, Andrey Temko, Climent Nadeu, Cristian Canton

Detection of acoustic events (AED) that take place in a meetingroom environment becomes a difficult task when signals show a large proportion of temporal overlap of sounds, like in seminartype data, where the acoustic events often occur simultaneously with speech. Whenever the event that produces the sound is related to a given position or movement, video signals may be a useful additional source of information for AED. In this work, we aim at improving the AED accuracy by using two complementary audio-based AED systems, built with SVM and HMM classifiers, and also a video-based AED system, which employs the output of a 3D video tracking algorithm to improve detection of steps. Fuzzy integral is used to fuse the outputs of the three classification systems in two stages. Experimental results using the CLEAR'07 evaluation data show that the detection rate increases by fusing the two audio information sources, and it is further improved by including video information.

doi: 10.21437/Interspeech.2008-28

Cite as: Butko, T., Temko, A., Nadeu, C., Canton, C. (2008) Fusion of audio and video modalities for detection of acoustic events. Proc. Interspeech 2008, 123-126, doi: 10.21437/Interspeech.2008-28

  author={Taras Butko and Andrey Temko and Climent Nadeu and Cristian Canton},
  title={{Fusion of audio and video modalities for detection of acoustic events}},
  booktitle={Proc. Interspeech 2008},