We present an evaluation of a Web-deployed spoken language CALL system, carried out using crowd-sourcing methods. The system, “Survival Japanese”, is a crash course in tourist Japanese implemented within the platform CALL-SLT. The evaluation was carried out over one week using the Amazon Mechanical Turk. Although we found a high proportion of attempted scammers, there was a core of 23 subjects who used the system in a responsible manner. The evidence that these subjects learned from their 111 sessions and 9092 spoken interactions was significant at P =0.001. Our conclusion is that crowd-sourcing is a potentially valid method for evaluating spoken CALL systems.
Index Terms. CALL, crowd-sourcing, speech recognition, evaluation, Japanese
Cite as: Rayner, M., Frank, I., Chua, C., Tsourakis, N., Bouillon, P. (2011) For a fistful of dollars: using crowd-sourcing to evaluate a spoken language CALL application. Proc. Speech and Language Technology in Education (SLaTE 2011), 117-120
@inproceedings{rayner11_slate,
author={Manny Rayner and Ian Frank and Cathy Chua and Nikos Tsourakis and Pierrette Bouillon},
title={{For a fistful of dollars: using crowd-sourcing to evaluate a spoken language CALL application}},
year=2011,
booktitle={Proc. Speech and Language Technology in Education (SLaTE 2011)},
pages={117--120}
}