ISCA Archive Interspeech 2021
ISCA Archive Interspeech 2021

Rethinking End-to-End Evaluation of Decomposable Tasks: A Case Study on Spoken Language Understanding

Siddhant Arora, Alissa Ostapenko, Vijay Viswanathan, Siddharth Dalmia, Florian Metze, Shinji Watanabe, Alan W. Black

Decomposable tasks are complex and comprise of a hierarchy of sub-tasks. Spoken intent prediction, for example, combines automatic speech recognition and natural language understanding. Existing benchmarks, however, typically hold out examples for only the surface-level sub-task. As a result, models with similar performance on these benchmarks may have unobserved performance differences on the other sub-tasks. To allow insightful comparisons between competitive end-to-end architectures, we propose a framework to construct robust test sets using coordinate ascent over sub-task specific utility functions. Given a dataset for a decomposable task, our method optimally creates a test set for each sub-task to individually assess sub-components of the end-to-end model. Using spoken language understanding as a case study, we generate new splits for the Fluent Speech Commands and Snips SmartLights datasets. Each split has two test sets: one with held-out utterances assessing natural language understanding abilities, and one with held-out speakers to test speech processing skills. Our splits identify performance gaps up to 10% between end-to-end systems that were within 1% of each other on the original test sets. These performance gaps allow more realistic and actionable comparisons between different architectures, driving future model development. We release our splits and tools for the community.


doi: 10.21437/Interspeech.2021-1537

Cite as: Arora, S., Ostapenko, A., Viswanathan, V., Dalmia, S., Metze, F., Watanabe, S., Black, A.W. (2021) Rethinking End-to-End Evaluation of Decomposable Tasks: A Case Study on Spoken Language Understanding. Proc. Interspeech 2021, 1264-1268, doi: 10.21437/Interspeech.2021-1537

@inproceedings{arora21_interspeech,
  author={Siddhant Arora and Alissa Ostapenko and Vijay Viswanathan and Siddharth Dalmia and Florian Metze and Shinji Watanabe and Alan W. Black},
  title={{Rethinking End-to-End Evaluation of Decomposable Tasks: A Case Study on Spoken Language Understanding}},
  year=2021,
  booktitle={Proc. Interspeech 2021},
  pages={1264--1268},
  doi={10.21437/Interspeech.2021-1537}
}