ISCA Archive Interspeech 2013
ISCA Archive Interspeech 2013

MINT.tools: tools and adaptors supporting acquisition, annotation and analysis of multimodal corpora

Spyros Kousidis, Thies Pfeiffer, David Schlangen

This paper presents a collection of tools (and adaptors for existing tools) that we have recently developed, which support acquisition, annotation and analysis of multimodal corpora. For acquisition, an extensible architecture is offered that integrates various sensors, based on existing connectors (e.g. for motion capturing via VICON, or ART) and on connectors we contribute (for motion tracking via Microsoft Kinect as well as eye tracking via Seeingmachines FaceLAB 5). The architecture provides live visualisation of the multimodal data in a unified virtual reality (VR) view (using Fraunhofer Instant Reality) for control during recordings, and enables recording of synchronised streams. For annotation, we provide a connection between the annotation tool ELAN (MPI Nijmegen) and the VR visualisation. For analysis, we provide routines in the programming language Python that read in and manipulate (aggregate, transform, plot, analyse) the sensor data, as well as text annotation formats (Praat TextGrids). Use of this toolset in multimodal studies proved to be efficient and effective, as we discuss. We make the collection available as open source for use by other researchers.


doi: 10.21437/Interspeech.2013-609

Cite as: Kousidis, S., Pfeiffer, T., Schlangen, D. (2013) MINT.tools: tools and adaptors supporting acquisition, annotation and analysis of multimodal corpora. Proc. Interspeech 2013, 2649-2653, doi: 10.21437/Interspeech.2013-609

@inproceedings{kousidis13b_interspeech,
  author={Spyros Kousidis and Thies Pfeiffer and David Schlangen},
  title={{MINT.tools: tools and adaptors supporting acquisition, annotation and analysis of multimodal corpora}},
  year=2013,
  booktitle={Proc. Interspeech 2013},
  pages={2649--2653},
  doi={10.21437/Interspeech.2013-609}
}