Leveraging Acoustic Cues and Paralinguistic Embeddings to Detect Expression from Voice

Vikramjit Mitra, Sue Booker, Erik Marchi, David Scott Farrar, Ute Dorothea Peitz, Bridget Cheng, Ermine Teves, Anuj Mehta, Devang Naik


Millions of people reach out to digital assistants such as Siri every day, asking for information, making phone calls, seeking assistance, and much more. The expectation is that such assistants should understand the intent of the user’s query. Detecting the intent of a query from a short, isolated utterance is a difficult task. Intent cannot always be obtained from speech-recognized transcriptions. A transcription-driven approach can interpret what has been said but fails to acknowledge how it has been said, and as a consequence, may ignore the expression present in the voice. Our work investigates whether a system can reliably detect vocal expression in queries using acoustic and paralinguistic embedding. Results show that the proposed method offers a relative equal error rate (EER) decrease of 60% compared to a bag-of-word based system, corroborating that expression is significantly represented by vocal attributes, rather than being purely lexical. Addition of emotion embedding helped to reduce the EER by 30% relative to the acoustic embedding, demonstrating the relevance of emotion in expressive voice.


 DOI: 10.21437/Interspeech.2019-2998

Cite as: Mitra, V., Booker, S., Marchi, E., Farrar, D.S., Peitz, U.D., Cheng, B., Teves, E., Mehta, A., Naik, D. (2019) Leveraging Acoustic Cues and Paralinguistic Embeddings to Detect Expression from Voice. Proc. Interspeech 2019, 1651-1655, DOI: 10.21437/Interspeech.2019-2998.


@inproceedings{Mitra2019,
  author={Vikramjit Mitra and Sue Booker and Erik Marchi and David Scott Farrar and Ute Dorothea Peitz and Bridget Cheng and Ermine Teves and Anuj Mehta and Devang Naik},
  title={{Leveraging Acoustic Cues and Paralinguistic Embeddings to Detect Expression from Voice}},
  year=2019,
  booktitle={Proc. Interspeech 2019},
  pages={1651--1655},
  doi={10.21437/Interspeech.2019-2998},
  url={http://dx.doi.org/10.21437/Interspeech.2019-2998}
}