ISCA International Workshop on Speech and Language Technology in Education (SLaTE 2011) Venice, Italy August 24-26, 2011

For A Fistful Of Dollars: Using Crowd-Sourcing To Evaluate A Spoken Language CALL Application

Manny Rayner (1), Ian Frank (2), Cathy Chua (3), Nikos Tsourakis (1), Pierrette Bouillon (1)

(1) University of Geneva, ETI/TIM/ISSCO, Geneva, Switzerland
(2) Future University Hakodate, Japan
(3) Faculty of Information and Communication Technologies, and Swinburne University of Technology, Melbourne, Australia

We present an evaluation of a Web-deployed spoken language CALL system, carried out using crowd-sourcing methods. The system, “Survival Japanese”, is a crash course in tourist Japanese implemented within the platform CALL-SLT. The evaluation was carried out over one week using the Amazon Mechanical Turk. Although we found a high proportion of attempted scammers, there was a core of 23 subjects who used the system in a responsible manner. The evidence that these subjects learned from their 111 sessions and 9092 spoken interactions was significant at P =0.001. Our conclusion is that crowd-sourcing is a potentially valid method for evaluating spoken CALL systems.
Index Terms. CALL, crowd-sourcing, speech recognition, evaluation, Japanese

Full Paper

Bibliographic reference.  Rayner, Manny / Frank, Ian / Chua, Cathy / Tsourakis, Nikos / Bouillon, Pierrette (2011): "For a fistful of dollars: using crowd-sourcing to evaluate a spoken language CALL application", In SLaTE-2011, 117-120.