ISCA Archive Interspeech 2021
ISCA Archive Interspeech 2021

Personalized Speech Enhancement Through Self-Supervised Data Augmentation and Purification

Aswin Sivaraman, Sunwoo Kim, Minje Kim

Training personalized speech enhancement models is innately a no-shot learning problem due to privacy constraints and limited access to noise-free speech from the target user. If there is an abundance of unlabeled noisy speech from the test-time user, one may train a personalized speech enhancement model using self-supervised learning. One straightforward approach to model personalization is to use the target speaker’s noisy recordings as pseudo-sources. Then, a pseudo denoising model learns to remove injected training noises and recover the pseudo-sources. However, this approach is volatile as it depends on the quality of the pseudo-sources, which may be too noisy. To remedy this, we propose a data purification step that refines the self-supervised approach. We first train an SNR predictor model to estimate the frame-by-frame SNR of the pseudo-sources. Then, we convert the predictor’s estimates into weights that adjust the pseudo-sources’ frame-by-frame contribution towards training the personalized model. We empirically show that the proposed data purification step improves the usability of the speaker-specific noisy data in the context of personalized speech enhancement. Our approach may be seen as privacy-preserving as it does not rely on any clean speech recordings or speaker embeddings.

doi: 10.21437/Interspeech.2021-1868

Cite as: Sivaraman, A., Kim, S., Kim, M. (2021) Personalized Speech Enhancement Through Self-Supervised Data Augmentation and Purification. Proc. Interspeech 2021, 2676-2680, doi: 10.21437/Interspeech.2021-1868

  author={Aswin Sivaraman and Sunwoo Kim and Minje Kim},
  title={{Personalized Speech Enhancement Through Self-Supervised Data Augmentation and Purification}},
  booktitle={Proc. Interspeech 2021},