Grad-StyleSpeech: Any-Speaker Adaptive Text-to-Speech Synthesis with Diffusion Models

By Minki Kang, Dong Min, Sung Ju Hwang. 2022

64 citations recorded by the source corpus.

There has been a significant progress in Text-To-Speech (TTS) synthesis technology in recent years, thanks to the advancement in neural generative modeling. However, existing methods on any-speaker adaptive TTS have achieved unsatisfactory performance, due to their suboptimal accuracy in mimicking the target speakers’ styles. In this work, we present Grad-StyleSpeech, which is an any-speaker adaptive TTS framework that is based on a diffusion model that can generate highly natural speech with extremely high similarity to target speakers’ voice, given a few seconds of reference speech. Grad-StyleSpeech significantly outperforms recent speaker-adaptive TTS baselines on English benchmarks. Audio samples are available at https://nardien.github.io/grad-stylespeech-demo.

Subfields: Artificial Intelligence, Generative Models, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Zhou Zhao
3. Shujie Liu
4. Yi Ren
5. Xu Tan
6. Zhuo Chen
7. Jinglin Liu
8. Sheng Zhao

In Generative Models

1. Zhou Zhao
2. Yi Ren
3. Rongjie Huang
4. Jinglin Liu
5. Zhuo Chen
6. Qiang Yang
7. Jinyu Li
8. Zhenhui Ye

In Speech & Audio

1. Zhou Zhao
2. Yi Ren
3. Jinglin Liu
4. Lei He
5. Rongjie Huang
6. Sheng Zhao
7. Yanqing Liu
8. Jinyu Li

In Signal Processing

1. Yi Ren
2. Jinyu Li
3. Zhou Zhao
4. Jinglin Liu
5. Rongjie Huang
6. Lei He
7. Shujie Liu
8. Sheng Zhao

Citations in the corpus

• Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers (2023, positive): “Additionally, diffusion model-based TTS [26], [27] has been extended to zero-shot TTS, yielding promising results [28].”
• Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias (2023, background): “Previous approaches can be categorized into speaker adaptation [13, 11, 60, 23] and speaker encoding [25, 1, 26, 61] methods.”
• A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AI (2023, unknown): “Also follows Grad-TTS [85], another work Grad-StyleSpeech [32] encodes the mel-spectrogram of reference speech to a styled vector, which is involved in the Manuscript submitted to ACM training of diffusion models.”
• Mega-TTS 2: Zero-Shot Text-to-Speech with Arbitrary Length Speech Prompts (2023, methodology): “Previous zero-shot TTS models introduce the timbre encoder with temporal average pooling before the output layer (e.g., Meta-stylespeech (Min et al., 2021), Grad-stylespeech (Kang et al., 2023), and Mega-TTS (Jiang et al., 2023)).”
• HierSpeech++: Bridging the Gap Between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-Shot Speech Synthesis (2023, unknown): “In this regard, it is di ffi cult to directly infer these rich representations from text by exacerbating the one-to-many mapping problem, and this may result in mispronunciation, oversmoothed speech, and a lack of similarity.”
• Faster Segment Anything: Towards Lightweight SAM for Mobile Applications (2023, background): “Beyond 2D vision, the investigation of SAM has also been extended to 3D object reconstruction Shen et al. [2023], Kang et al. [2022], demonstrating its capabilities in assisting 3D model generation from a single image.”