In this paper we describe a human benchmark experiment for language recognition. We used the same task, data and evaluation measure as in the NIST Language Recognition Evaluation (LRE) 2005. For the primary condition of interest all 10-second trials were used in the experiment. The experiment was conducted by 38 subjects, who each processed part of the trials. For the seven-language closed set condition the human subjects obtained an average CDET of 23.1%. This result can be compared to machine results of the 2005 submission, for instance that of Brno University of Technology, whose system scored 7.15% at this task. A detailed statistical analysis is given of the human benchmark results. We argue that the result can best be expressed as the performance of naive subjects.
Cite as: Leeuwen, D.A.v., Boer, M.d., Orr, R. (2008) A human benchmark for the NIST language recognition evaluation 2005. Proc. The Speaker and Language Recognition Workshop (Odyssey 2008), paper 12
@inproceedings{leeuwen08_odyssey,
author={David A. van Leeuwen and Michael de Boer and Rosemary Orr},
title={{A human benchmark for the NIST language recognition evaluation 2005}},
year=2008,
booktitle={Proc. The Speaker and Language Recognition Workshop (Odyssey 2008)},
pages={paper 12}
}