16th Annual Conference of the International Speech Communication Association

Dresden, Germany
September 6-10, 2015

Real-Time Audio-to-Score Alignment of Singing Voice Based on Melody and Lyric Information

Rong Gong, Philippe Cuvillier, Nicolas Obin, Arshia Cont

IRCAM, France

Singing voice is specific in music: a vocal performance conveys both music (melody/pitch) and lyrics (text/phoneme) content. This paper aims at exploiting the advantages of melody and lyric information for real-time audio-to-score alignment of singing voice. First, lyrics are added as a separate observation stream into a template-based hidden semi-Markov model (HSMM), whose observation model is based on the construction of vowel templates. Second, early and late fusion of melody and lyric information are processed during real-time audio-to-score alignment. An experiment conducted with two professional singers (male/female) shows that the performance of a lyrics-based system is comparable to that of melody-based score following systems. Furthermore, late fusion of melody and lyric information substantially improves the alignment performance. Finally, maximum a posteriori adaptation (MAP) of the vowel templates from one singer to the other suggests that lyric information can be efficiently used for any singer.

Full Paper

Bibliographic reference.  Gong, Rong / Cuvillier, Philippe / Obin, Nicolas / Cont, Arshia (2015): "Real-time audio-to-score alignment of singing voice based on melody and lyric information", In INTERSPEECH-2015, 3312-3316.