DiffVoice: Text-to-Speech with Latent Diffusion
Online Supplement

Abstract

In this work, we present DiffVoice, a novel text to speech model based on latent diffusion. We propose to first encode speech signals into a phoneme rate latent representation with a variational autoencoder enhanced by adversarial training, and then jointly model the duration and the latent representation with a diffusion model. Subjective evaluations demonstrate that our method beats the best publicly available systems in naturalness. By adopting recent generative inverse problem solving algorithms for diffusion models, DiffVoice achieves the state of the art performance in text based speech editing, and zero shot adaptation.

The following are 6 different renditions of the abstract by a DiffVoice model. Note that this model does not have any speaker input.It samples DIFFerent VOICEs from the latent space.

This notion page contains a few more audio samples using the BigVGAN vocoder.

Results for Seen Speakers in LibriTTS

20 random speakers seen during training from the train-clean-100 split of LibriTTS.

Zeroshot Synthesis on LibriTTS test-clean

All speakers are unseen from the test-clean split of LibriTTS.

Zeroshot Synthesis on VCTK

All speakers are unseen from VCTK

Speech Editing with DiffVoice

In this section, we demonstrate DiffVoice's ability to conduct text-based speech editing (including insertion, replacement, and inpainting) with state-of-the-art quality.

The following audio samples are obtained from RetrieverTTS website. We specially thank the authors of RetrieverTTS for providing their generated samples for evaluation.

(Speech Inpainting) Ruth was glad to hear that Philip had made a push into the world, and she was sure that his talent and courage would make a way for him. She should pray for his success at any rate, and especially that the Indians, in St. Louis, would not take his scalp.

(Speech Insertion) Is the under side of civilization any less important than the upper side of the very same civilization merely because it is deeper and more sombre?

(Speech Insertion) They are chiefly formed from combinations under his success of the impressions made in childhood.

The following samples are taken from the website of EditSpeech. To which we have no access of the implementation. EditSpeech is trained on VCTK, instead of LibriTTS.

(Speech Insertion) some have accepted it as a miracle never seen before without physical explanation

(Speech Replacement) some have accepted it as a miracle an undeniable fact without physical explanation

(Speech Insertion) for that theoretical and realistic reason cover should not be given

(Speech Replacement) you should always be able to get out in focus your research on some direction

Extra: Comparison with GuidedTTS 2

In this section, we compare our model with Guided TTS 2 by comparing samples obtained from its demo page. GuidedTTS 2 samples are downsampled to 16kHz for comparison. Compared with our work, GuidedTTS 2 is trained on a much larger dataset, and it does not require transcriptions of the source audios.the transcription of the source audios.