ISCA Archive Interspeech 2009
ISCA Archive Interspeech 2009

Evaluating evaluators: a case study in understanding the benefits and pitfalls of multi-evaluator modeling

Emily Mower, Maja J. Matarić, Shrikanth S. Narayanan

Emotion perception is a complex process, often measured using stimuli presentation experiments that query evaluators for their perceptual ratings of emotional cues. These evaluations contain large amounts of variability both related and unrelated to the evaluated utterances. One approach to handling this variability is to model emotion perception at the individual level. However, the perceptions of specific users may not adequately capture the emotional acoustic properties of an utterance. This problem can be mitigated by the common technique of averaging evaluations from multiple users. We demonstrate that this averaging procedure improves classification performance when compared to classification results from models created using individual-specific evaluations. We also demonstrate that the performance increases are related to the consistency with which evaluators label data. These results suggest that the acoustic properties of emotional speech are better captured using models formed from averaged evaluations rather than from individual-specific evaluations.


doi: 10.21437/Interspeech.2009-471

Cite as: Mower, E., Matarić, M.J., Narayanan, S.S. (2009) Evaluating evaluators: a case study in understanding the benefits and pitfalls of multi-evaluator modeling. Proc. Interspeech 2009, 1583-1586, doi: 10.21437/Interspeech.2009-471

@inproceedings{mower09_interspeech,
  author={Emily Mower and Maja J. Matarić and Shrikanth S. Narayanan},
  title={{Evaluating evaluators: a case study in understanding the benefits and pitfalls of multi-evaluator modeling}},
  year=2009,
  booktitle={Proc. Interspeech 2009},
  pages={1583--1586},
  doi={10.21437/Interspeech.2009-471}
}