 |
ISCA International Workshop on Speech and Language Technology in Education (SLaTE 2009)Wroxall Abbey Estate, Warwickshire, England |
 |
|
ISCA International Workshop on Speech and Language Technology in Education (SLaTE 2009)Wroxall Abbey Estate, Warwickshire, England |
|
This paper presents an analysis of differences in human transcriptions of non-native
spontaneous speech on a
word level, collected in the context of an English Proficiency Test. While transcribers of
native speech
typically agree at a very high level (5% word error rate or less), this study finds
substantially higher
disagreement rates between transcribers of non-native speech (10%-34% word error rate).
We show how
transcription disagreements are negatively correlated to the length of utterances (fewer
contexts) and to
human scores (impact of lower speaker proficiency) and also seem to be affected by the
audio quality of
the recordings.
We also demonstrate how a novel multi-stage transcription procedure
using selection and
ranking of transcription alternatives by peers can achieve a higher quality gold standard
that approaches
the quality of native speech transcription.
Bibliographic reference. Zechner, Klaus (2009): "What did they actually say? agreement and disagreement among transcribers of non-native spontaneous speech responses in an English proficiency test", In SLaTE-2009, 25-28.