End-to-End ASR with Adaptive Span Self-Attention

Xuankai Chang, Aswin Shanmugam Subramanian, Pengcheng Guo, Shinji Watanabe, Yuya Fujita, Motoi Omachi


Transformers have demonstrated state-of-the-art performance on many tasks in natural language processing and speech processing. One of the key components in Transformers is self-attention, which attends to the whole input sequence at every layer. However, the computational and memory cost of self-attention is square of the input sequence length, which is a major concern in automatic speech recognition (ASR) where the input sequence can be very long. In this paper, we propose to use a technique called adaptive span self-attention for ASR tasks, which is originally proposed for language modeling. Our method enables the network to learn an appropriate size and position of the window for each layer and head, and our newly introduced scheme can further control the window size depending on the future and past contexts. Thus, it can save both computational complexity and memory size from the square order of the input length to the adaptive linear order. We show the effectiveness of the proposed method by using several ASR tasks, and the proposed adaptive span methods consistently improved the performance from the conventional fixed span methods.


 DOI: 10.21437/Interspeech.2020-2816

Cite as: Chang, X., Subramanian, A.S., Guo, P., Watanabe, S., Fujita, Y., Omachi, M. (2020) End-to-End ASR with Adaptive Span Self-Attention. Proc. Interspeech 2020, 3595-3599, DOI: 10.21437/Interspeech.2020-2816.


@inproceedings{Chang2020,
  author={Xuankai Chang and Aswin Shanmugam Subramanian and Pengcheng Guo and Shinji Watanabe and Yuya Fujita and Motoi Omachi},
  title={{End-to-End ASR with Adaptive Span Self-Attention}},
  year=2020,
  booktitle={Proc. Interspeech 2020},
  pages={3595--3599},
  doi={10.21437/Interspeech.2020-2816},
  url={http://dx.doi.org/10.21437/Interspeech.2020-2816}
}