ISCA Archive SSW 2021
ISCA Archive SSW 2021

Low-latency real-time non-parallel voice conversion based on cyclic variational autoencoder and multiband WaveRNN with data-driven linear prediction

Patrick Lumban Tobing, Tomoki Toda

This paper presents a low-latency real-time (LLRT) non-parallel voice conversion (VC) framework based on cyclic variational autoencoder (CycleVAE) and multiband WaveRNN with datadriven linear prediction (MWDLP). CycleVAE is a robust nonparallel multispeaker spectral model, which utilizes a speakerindependent latent space and a speaker-dependent code to generate reconstructed/converted spectral features given the spectral features of an input speaker. On the other hand, MWDLP is an efficient and a high-quality neural vocoder that can handle multispeaker data and generate speech waveform for LLRT applications with CPU. To accommodate LLRT constraint with CPU, we propose a novel CycleVAE framework that utilizes mel-spectrogram as spectral features and is built with a sparse network architecture. Further, to improve the modeling performance, we also propose a novel fine-tuning procedure that refines the frame-rate CycleVAE network by utilizing the waveform loss from the MWDLP network. The experimental results demonstrate that the proposed framework achieves highperformance VC, while allowing for LLRT usage with a singlecore of 2.1–2.7 GHz CPU on a real-time factor of 0.87–0.95, including input/output, feature extraction, on a frame shift of 10 ms, a window length of 27.5 ms, and 2 lookup frames.


doi: 10.21437/SSW.2021-25

Cite as: Tobing, P.L., Toda, T. (2021) Low-latency real-time non-parallel voice conversion based on cyclic variational autoencoder and multiband WaveRNN with data-driven linear prediction. Proc. 11th ISCA Speech Synthesis Workshop (SSW 11), 142-147, doi: 10.21437/SSW.2021-25

@inproceedings{tobing21_ssw,
  author={Patrick Lumban Tobing and Tomoki Toda},
  title={{Low-latency real-time non-parallel voice conversion based on cyclic variational autoencoder and multiband WaveRNN with data-driven linear prediction}},
  year=2021,
  booktitle={Proc. 11th ISCA Speech Synthesis Workshop (SSW 11)},
  pages={142--147},
  doi={10.21437/SSW.2021-25}
}