ISCA Archive Interspeech 2008
ISCA Archive Interspeech 2008

Assessing agreement of observer- and self-annotations in spontaneous multimodal emotion data

Khiet P. Truong, Mark A. Neerincx, David A. van Leeuwen

We investigated inter-observer agreement and the reliability of self-reported emotion ratings (i.e., self-raters judging their own emotions) in spontaneous multimodal emotion data. During a multiplayer video game, vocal and facial expressions were recorded (including the game content itself) and were annotated by the players themselves on arousal and valence scales. In a perception experiment, observers rated a small part of the data that was provided in 4 conditions: audio only, visual only, audiovisual and audiovisual plus context. Inter-observer agreements varied between 0.32 and 0.52 when the ratings were scaled. Providing multimodal information usually increased agreement. Finally, we found that the averaged agreement between the self-rater and the observers was somewhat lower than the inter-observer agreement.


doi: 10.21437/Interspeech.2008-95

Cite as: Truong, K.P., Neerincx, M.A., Leeuwen, D.A.v. (2008) Assessing agreement of observer- and self-annotations in spontaneous multimodal emotion data. Proc. Interspeech 2008, 318-321, doi: 10.21437/Interspeech.2008-95

@inproceedings{truong08_interspeech,
  author={Khiet P. Truong and Mark A. Neerincx and David A. van Leeuwen},
  title={{Assessing agreement of observer- and self-annotations in spontaneous multimodal emotion data}},
  year=2008,
  booktitle={Proc. Interspeech 2008},
  pages={318--321},
  doi={10.21437/Interspeech.2008-95}
}