SpeechPainter: Text-conditioned Speech Inpainting

By Zalán Borsos, Matthew Sharifi, Marco Tagliasacchi. 2022

36 citations recorded by the source corpus.

We propose SpeechPainter, a model for filling in gaps of up to one second in speech samples by leveraging an auxiliary textual input. We demonstrate that the model performs speech inpainting with the appropriate content, while maintaining speaker identity, prosody and recording environment conditions, and generalizing to unseen speakers. Our approach significantly outperforms baselines constructed using adaptive TTS, as judged by human raters in side-by-side preference and MOS tests.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Wei-Ning Hsu
2. Yossi Adi
3. Yiwei Guo
4. Sheng Zhao
5. Yanqing Liu
6. Bowen Shi
7. K. Yu
8. Apoorv Vyas

Citations in the corpus

• Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale (2023, methodology): “While this is a step toward building large scale generalist models using little explicit supervision, most prior work using text guidance still assumes a deterministic mapping from text and context to target [3, 5], which is only realistic for very short segments.”
• RetrieverTTS: Modeling Decomposed Factors for Text-Based Speech Insertion (2022, methodology): “Two existing works [4, 5] are directly aimed at this task.”
• AUDIT: Audio Editing by Following Instructions with Latent Diffusion Models (2023, unknown): “[36, 5, 33, 55] explored the task of speech/music inpainting based on the corresponding text or music score.”
• DiffVoice: Text-to-Speech with Latent Diffusion (2023, methodology): “We demonstrate that DiffVoice can achieve state-of-the-art performance on this task, without specially tailored model designs and training procedures adopted by many prior works [16– 19].”