Auxiliary Interference Speaker Loss for Target-Speaker Speech Recognition

Naoyuki Kanda, Shota Horiguchi, Ryoichi Takashima, Yusuke Fujita, Kenji Nagamatsu, Shinji Watanabe


In this paper, we propose a novel auxiliary loss function for target-speaker automatic speech recognition (ASR). Our method automatically extracts and transcribes target speaker’s utterances from a monaural mixture of multiple speakers speech given a short sample of the target speaker. The proposed auxiliary loss function attempts to additionally maximize interference speaker ASR accuracy during training. This will regularize the network to achieve a better representation for speaker separation, thus achieving better accuracy on the target-speaker ASR. We evaluated our proposed method using two-speaker-mixed speech in various signal-to-interference-ratio conditions. We first built a strong target-speaker ASR baseline based on the state-of-the-art lattice-free maximum mutual information. This baseline achieved a word error rate (WER) of 18.06% on the test set while a normal ASR trained with clean data produced a completely corrupted result (WER of 84.71%). Then, our proposed loss further reduced the WER by 6.6% relative to this strong baseline, achieving a WER of 16.87%. In addition to the accuracy improvement, we also showed that the auxiliary output branch for the proposed loss can even be used for a secondary ASR for interference speakers’ speech.


 DOI: 10.21437/Interspeech.2019-1126

Cite as: Kanda, N., Horiguchi, S., Takashima, R., Fujita, Y., Nagamatsu, K., Watanabe, S. (2019) Auxiliary Interference Speaker Loss for Target-Speaker Speech Recognition. Proc. Interspeech 2019, 236-240, DOI: 10.21437/Interspeech.2019-1126.


@inproceedings{Kanda2019,
  author={Naoyuki Kanda and Shota Horiguchi and Ryoichi Takashima and Yusuke Fujita and Kenji Nagamatsu and Shinji Watanabe},
  title={{Auxiliary Interference Speaker Loss for Target-Speaker Speech Recognition}},
  year=2019,
  booktitle={Proc. Interspeech 2019},
  pages={236--240},
  doi={10.21437/Interspeech.2019-1126},
  url={http://dx.doi.org/10.21437/Interspeech.2019-1126}
}