Odyssey 2008: The Speaker and Language Recognition Workshop

Stellenbosch, South Africa
January 21-24, 2008

A Human Benchmark for the NIST Language Recognition Evaluation 2005

David A. van Leeuwen (1), Michael de Boer (1,2), Rosemary Orr (2)

(1) TNO Human Factors, Soesterberg, the Netherlands
(2) University College Utrecht, Utrecht, The Netherlands

In this paper we describe a human benchmark experiment for language recognition. We used the same task, data and evaluation measure as in the NIST Language Recognition Evaluation (LRE) 2005. For the primary condition of interest all 10-second trials were used in the experiment. The experiment was conducted by 38 subjects, who each processed part of the trials. For the seven-language closed set condition the human subjects obtained an average CDET of 23.1%. This result can be compared to machine results of the 2005 submission, for instance that of Brno University of Technology, whose system scored 7.15% at this task. A detailed statistical analysis is given of the human benchmark results. We argue that the result can best be expressed as the performance of ‘naive subjects.’

Full Paper

Bibliographic reference.  Leeuwen, David A. van / Boer, Michael de / Orr, Rosemary (2008): "A human benchmark for the NIST language recognition evaluation 2005", In Odyssey-2008, paper 012.