Third International Conference on Spoken Language Processing (ICSLP 94)

Yokohama, Japan
September 18-22, 1994

Evaluation of Prosodic Transcription Labeling Reliability in The Tobi Framework

John F. Pitrelli (1), Mary E. Beckman (2), Julia Hirschberg (3)

(1) NYNEX Science & Technology, Inc., White Plains, NY, USA
(2) Ohio State University, USA
(3) AT&T Bell Laboratories, USA

A diverse group of speech scientists and engineers has developed the ToBI (TOnes and Break Indices) prosodic transcription system and materials to teach it to transcribers. ToBI consists of parallel tiers reflecting the multiple components of prosody, the most important being a tone tier, for intonational analysis, and a break index tier, for indicating strength of coherence or disjunctive between adjacent words. To assess the system, we measured inter-transcriber agreement on utterances representative of the varied types of speech important to researchers, employing a diverse set of transcribers ranging from experts to newly-trained users. Consistency was measured in terms of number of transcriber pairs agreeing on the labeling of each particular word, a stringent metric. Using this metric, we observe 88% agreement on the presence or absence of a particular category of tonal element, and 81% agreement on the exact label for a tonal category. For break indices, agreement to within one level occurs 92% of the time. We conclude that the ToBI standard and its training materials have been refined to the point that they can be used fruitfully for largescale annotation of prosodic phenomena in speech databases.

Full Paper

Bibliographic reference.  Pitrelli, John F. / Beckman, Mary E. / Hirschberg, Julia (1994): "Evaluation of prosodic transcription labeling reliability in the tobi framework", In ICSLP-1994, 123-126.