ViVoLAB Multimodal Diarization System for RTVE 2020 Challenge

Victoria Mingote, Ignacio Viñals, Pablo Gimeno, Antonio Miguel, Alfonso Ortega, Eduardo Lleida

This paper describes a post-evaluation analysis of the system developed by ViVoLAB research group for the IberSPEECH-RTVE 2020 Multimodal Diarization (MD) Challenge. This challenge is focused on the study of multimodal systems for the diarization of audiovisual files and the assignment of an identity to each segment. In this work, we have implemented two different subsystems to address this task using the images and the audio from files separately. To develop our subsystems, we have employed the state of the art speaker and face verification embeddings extracted from publicly available Deep Neural Networks (DNN). Different clustering approaches are also used in combination with the tracking and identity assignment process. Furthermore, in the face verification system, we have included a novel approach to train an enrollment model for each identity which we have shown previously to improve the results compared to the average of the enrollment data. Using this approach, we train a learnable vector to represent each enrollment character.

doi: 10.21437/IberSPEECH.2021-16

Mingote, V, Viñals, I, Gimeno, P, Miguel, A, Ortega, A, Lleida, E (2021) ViVoLAB Multimodal Diarization System for RTVE 2020 Challenge. Proc. IberSPEECH 2021, 76-80, doi: 10.21437/IberSPEECH.2021-16.