AdaSpeech: Adaptive Text to Speech for Custom Voice

By Mingjian Chen, Xu Tan, Bohan Li, Yanqing Liu, Tao Qin, Sheng Zhao, Tie-Yan Liu. 2021

230 citations recorded by the source corpus.

Custom voice, a specific text to speech (TTS) service in commercial speech platforms, aims to adapt a source TTS model to synthesize personal voice for a target speaker using few speech data. Custom voice presents two unique challenges for TTS adaptation: 1) to support diverse customers, the adaptation model needs to handle diverse acoustic conditions that could be very different from source speech data, and 2) to support a large number of customers, the adaptation parameters need to be small enough for each target speaker to reduce memory usage while maintaining high voice quality. In this work, we propose AdaSpeech, an adaptive TTS system for high-quality and efficient customization of new voices. We design several techniques in AdaSpeech to address the two challenges in custom voice: 1) To handle different acoustic conditions, we use two acoustic encoders to extract an utterance-level vector and a sequence of phoneme-level vectors from the target speech during training; in inference, we extract the utterance-level vector from a reference speech and use an acoustic predictor to predict the phoneme-level vectors. 2) To better trade off the adaptation parameters and voice quality, we introduce conditional layer normalization in the mel-spectrogram decoder of AdaSpeech, and fine-tune this part in addition to speaker embedding for adaptation. We pre-train the source TTS model on LibriTTS datasets and fine-tune it on VCTK and LJSpeech datasets (with different acoustic conditions from LibriTTS) with few adaptation data, e.g., 20 sentences, about 1 minute speech. Experiment results show that AdaSpeech achieves much better adaptation quality than baseline methods, with only about 5K specific parameters for each speaker, which demonstrates its effectiveness for custom voice. Audio samples are available at https://speechresearch.github.io/adaspeech/.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Zhou Zhao
5. Yi Ren
6. Sheng Zhao
7. Jinglin Liu
8. Rongjie Huang

In Speech & Audio

1. Xu Tan
2. Yi Ren
3. Zhou Zhao
4. Tao Qin
5. Jinglin Liu
6. Sheng Zhao
7. Tie-Yan Liu
8. Rongjie Huang

In Signal Processing

1. Xu Tan
2. Yi Ren
3. Tao Qin
4. Zhou Zhao
5. Tie-Yan Liu
6. Jinglin Liu
7. Rongjie Huang
8. Sheng Zhao

Citations in the corpus

• Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers (2023, neutral): “Subsequent research in speaker adaptation has focused on improving efficiency, requiring less data and fewer speaker-specific parameters [6], [7], [20].”
• NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (2024, methodology): “We first add the representation of prosody, content, and acoustic details together and then fuse the timbre information by conditional layer normalization [45] to obtain the input z for the speech decoder.”
• Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias (2023, background): “Previous approaches can be categorized into speaker adaptation [13, 11, 60, 23] and speaker encoding [25, 1, 26, 61] methods.”
• Grad-StyleSpeech: Any-Speaker Adaptive Text-to-Speech Synthesis with Diffusion Models (2022, methodology): “Some of previous works [13, 16] need a few transcribed (supervised) samples for fine-tuning TTS model.”
• AdaSpeech 4: Adaptive Text to Speech in Zero-Shot Scenarios (2022, methodology): “Figure 1: The architecture of AdaSpeech 4, where the basic model backbone is based on AdaSpeech [10], but with three systematic designs to improve the generation ability on speaker characteristics in zero-shot scenarios.”
• Meta-TTS: Meta-Learning for Few-Shot Speaker Adaptive Text-to-Speech (2021, methodology): “AdaSpeech [11] uses FastSpeech2 as its architecture with two model modifications for the speaker adaptation approach: acoustic condition modeling and conditional LayerNorm.”
• A Survey on Neural Speech Synthesis (2021, methodology): “Adaptive General Adaptation AdaSpeech 1/3 [39, 398], [56, 260, 120, 37, 206]”
• FastSpeech 2: Fast and High-Quality End-to-End Text to Speech (2020, background): “TTS is a typical one-to-many mapping problem (Wang et al., 2017; Zhu et al., 2017; Jayne et al., 2012; Gadermayr et al., 2020; Chen et al., 2021), since multiple possible speech sequences can correspond to a text sequence due to variations in speech, such as pitch, duration, sound volume and prosody”
• Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models (2023, background): “Adapting models (Chen et al., 2020b; Huang et al., 2022) to a specific individual or object is a long-standing goal in machine learning research.”
• ProDiff: Progressive Fast Diffusion Model for High-Quality Text-to-Speech (2022, methodology): “Following the common practice [1, 28], we conduct preprocessing on the speech and text data: 1) extract the spectrogram with the FFT size of 1024, hop size of 256, and window size of 1024 samples; 2) convert it to a mel-spectrogram with 80 frequency bins; and 3) extract F0 (fundamental frequency) fr”
• Adaspeech 2: Adaptive Text to Speech with Untranscribed Data (2021, methodology): “Most previous works [1, 3, 4, 8] on TTS adaptation require paired data (speech and its transcripts), such as AdaSpeech [8].”
• DelightfulTTS: The Microsoft Speech Synthesis System for Blizzard Challenge 2021 (2021, methodology): “Additionally, we introduce an improved Conformer [2] block to better model the local and global dependency in acoustic model [3, 4].”
• Meta-StyleSpeech : Multi-Speaker Adaptive Text-to-Speech Generation (2021, background): “A popular approach is to train the model on a large multi-speakers dataset and then fine-tune the whole model (Chen et al., 2019; Arik et al., 2018) or only parts of the model (Moss et al., 2020; Zhang et al., 2020; Chen et al., 2021).”
• Probabilistic weather forecasting with machine learning (2024, unknown): “Conditioning on noise level σ is achieved by replacing all layer-norm layers in the architecture with conditional layer-norm 57 layers.”
• NANSY++: Unified Voice Synthesis with Neural Analysis and Synthesis (2022, background): “Another issue is that generating high-fidelity singing voice from mel spectrogram is well known to be difficult owing to a wide pitch range, long continuous pronunciation, and high sampling rate (Chen et al., 2020a; Perrotin et al., 2021; Morrison et al., 2022).”
• When large language models meet personalization: perspectives of challenges and opportunities (2023, unknown): “The multi-modal generation has become an essential aspect of AIGC, which learns the multimodal connection and interaction, typically including vision language generation [259], text audio generation [260], text graph generation [261], text Code Generation [262].”