This paper reports labeler agreement in the transcription of Korean prosody using Korean ToBI (K-ToBI). Twenty utterances representing five different types of speech were produced by 18 speakers and transcribed by 21 labelers differing in their levels of experience with K-ToBI. Following the stringent metric used for English ToBI evaluation [14,12], consistency was measured in terms of the number of transcriber pairs agreeing on the labeling of each particular word. The results show that for tonal transcriptions of the 32,130 transcriber-pair-words, agreement was 77% for the type of boundaries at the end of each word (i.e., word, AP, or IP), 78% for AP boundaries, and 91% for IP boundaries. For break indices, the agreement score for exact matching in the labeling was 59%, 69% when relaxing the presence/absence of diacritics, and 99% when relaxing within +/-1 level. In sum, the data confirm that the conventions of K-ToBI are adequate, easy to learn, and can be reliably used for research in Korean prosody and for large-scale prosodic annotation in speech databases.
Cite as: Jun, S.-A., Lee, S.-H., Kim, K., Lee, Y.-J. (2000) Labeler agreement in transcribing korean intonation with K-toBI. Proc. 6th International Conference on Spoken Language Processing (ICSLP 2000), vol. 3, 211-214
@inproceedings{jun00_icslp,
author={Sun-Ah Jun and Sook-Hyang Lee and Keeho Kim and Yong-Ju Lee},
title={{Labeler agreement in transcribing korean intonation with K-toBI}},
year=2000,
booktitle={Proc. 6th International Conference on Spoken Language Processing (ICSLP 2000)},
pages={vol. 3, 211-214}
}