16th Annual Conference of the International Speech Communication Association

Dresden, Germany
September 6-10, 2015

CapCap: An Output-Agreement Game for Video Captioning

Hernisa Kacorri (1), Kaoru Shinkawa (2), Shin Saito (2)

(1) CUNY Graduate Center, USA
(2) IBM Research Tokyo, Japan

CapCap is an output-agreement game that challenges players' listening and speaking skills. Players submit their transcriptions for short video segments against a countdown timer, in one of three pre-specified modes, to score points and support their team. Adding entertainment value, the game channels input toward captioning videos without monetary rewards. It deploys a novel human computation algorithm, which collects input from a crowd of non-experts, sequentially and in parallel, until a completion criterion is met. Rather than monetary incentive, CapCap uses motivational mechanisms like indirect feedback, mix of player skills, and community identification. Preliminary results from a field trial with mostly non-native English speakers improved the WER of English captions over ASR output.

Full Paper

Bibliographic reference.  Kacorri, Hernisa / Shinkawa, Kaoru / Saito, Shin (2015): "Capcap: an output-agreement game for video captioning", In INTERSPEECH-2015, 2814-2818.