Sixth International Conference on Spoken Language Processing
(ICSLP 2000)

Beijing, China
October 16-20, 2000

Inter-Transcriber Reliability of ToBI Prosodic Labeling

Ann K. Syrdal (1), Julia McGory (2)

(1) AT&T Labs - Research, Florham Park, NJ, USA
(2)Dept. of Linguistics, Ohio State University, Columbus, OH, USA

The goal of this study was to evaluate the reliability among transcribers of a standard prosodic labeling system under relatively optimal conditions of training, supervision, facilities, procedures, and extent of speaker familiarity. The ToBI (Tones and Break Indices) model for standard American English was used in the study; break indices indicate the degree of junction between words, pitch accents designate word prominence, and edge tones mark phrase boundaries. The American English speech corpora were read by a female professional speaker and by a male professional speaker, and were composed of several types of texts to ensure prosodic variety. Each of four experienced transcribers independently labeled each corpus. For each corpus, word level agreement in break indices, pitch accents, and edge tones between all possible pairs of transcribers was analyzed, and various statistics were calculated. Agreement among labelers was generally higher than that reported in previous studies[1,2] of larger and more diverse groups of labelers. Agreement was high for some prosodic categories, but low for others. The extent of reliability for various prosodic distinctions has important implications for refining the ToBI model and for limitations in the use of prosody in speech technologies.


  1. J. Pitrelli, M. Beckman, and J. Hirschberg. Evaluation of prosodic transcription labeling reliability in the ToBI framework. In Proc. 3rd Internat. Conf. Spoken Language Processing, volume 2, pages 123{126, Yokohama, 1994. ICSLP
  2. M. Grice, M. Reyelt, R. Benzmuller, J. Mayer, and A. Batliner. Consistency in transcription and labelling of German intonation with GToBI. In Proc. 4th Inter- nat. Conf. Spoken Language Processing, volume 3, pages 1716{1719, Philadelphia, 1996. ICSLP

Full Paper

Bibliographic reference.  Syrdal, Ann K. / McGory, Julia (2000): "Inter-transcriber reliability of toBI prosodic labeling", In ICSLP-2000, vol.3, 235-238.