EdiTTS: Score-based Editing for Controllable Text-to-Speech

By Jaesung Tae, Hyeongju Kim, Taesu Kim. 2021

51 citations recorded by the source corpus.

We present EdiTTS, an off-the-shelf speech editing methodology based on score-based generative modeling for text-to-speech synthesis. EdiTTS allows for targeted, granular editing of audio, both in terms of content and pitch, without the need for any additional training, task-specific optimization, or architectural modifications to the score-based model backbone. Specifically, we apply coarse yet deliberate perturbations in the Gaussian prior space to induce desired behavior from the diffusion model while applying masks and softening kernels to ensure that iterative edits are applied only to the target region. Through listening tests and speech-to-text back transcription, we show that EdiTTS outperforms existing baselines and produces robust samples that satisfy user-imposed requirements.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Ming-Hsuan Yang
2. Pei Sun
3. Bin Cui
4. Yue Zhao
5. Shoufa Chen
6. Wentao Zhang
7. Shenda Hong
8. Ling Yang

Citations in the corpus

• UniAudio: An Audio Foundation Model Toward Universal Audio Generation (2023, neutral): “…♣♢ (Wu et al., 2016) reverberant speech speech Instruct TTS (I-TTS) ♢ (Guo et al., 2023) phoneme, textual instruction speech Speech Edit (S-Edit) (Tae et al., 2021) ♢ phoneme (with duration), original speech speech continuous embeddings derived from pre-trained text large language models (Raffel…”
• UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocoding (2023, neutral): “In addition to speech continuation, there is another context-aware TTS task called speech editing (Tae, Kim, and Kim 2022; Yin et al. 2022).”
• Diffusion Models: A Comprehensive Survey of Methods and Applications (2022, methodology): “EdiTTS [228] leverages the score-based text-to-speech model to refine a mel-spectrogram prior that is coarsely modified.”
• DiffusionDet: Diffusion Model for Object Detection (2022, background): “…models have recently demonstrated remarkable results in fields including computer vision [4,21,33,35,39,68,71,75,76,82,104,108], nature language processing [3,30,52], audio processing [41,48,50,70, 91,100,103], graph-related topics [42], interdisciplinary applications [2,40,44,78,94,99,102], etc.”