Adaspeech 2: Adaptive Text to Speech with Untranscribed Data

By Yuzi Yan, Xu Tan, Bohan Li, Tao Qin, Sheng Zhao, Yuan Shen, Tie-Yan Liu. 2021

50 citations recorded by the source corpus.

Text to speech (TTS) is widely used to synthesize personal voice for a target speaker, where a well-trained source TTS model is fine-tuned with few paired adaptation data (speech and its transcripts) on this target speaker. However, in many scenarios, only untranscribed speech data is available for adaptation, which brings challenges to the previous TTS adaptation pipelines (e.g., AdaSpeech). In this paper, we develop AdaSpeech 2, an adaptive TTS system that only leverages untranscribed speech data for adaptation. Specifically, we introduce a mel-spectrogram encoder to a well-trained TTS model to conduct speech reconstruction, and at the same time constrain the output sequence of the mel-spectrogram encoder to be close to that of the original phoneme encoder. In adaptation, we use untranscribed speech data for speech reconstruction and only fine-tune the TTS decoder. AdaSpeech 2 has two advantages: 1) Pluggable: our system can be easily applied to existing trained TTS models without re-training. 2) Effective: our system achieves on-par voice quality with the transcribed TTS adaptation (e.g., AdaSpeech) with the same amount of untranscribed data, and achieves better voice quality than previous untranscribed adaptation methods 1.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Sheng Zhao
5. Frank K. Soong
6. Lei He
7. Yanqing Liu
8. Yi Ren

In Speech & Audio

1. Xu Tan
2. Sheng Zhao
3. Tao Qin
4. Tie-Yan Liu
5. Lei He
6. Yanqing Liu
7. Frank K. Soong
8. Bohan Li

In Signal Processing

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Sheng Zhao
5. Lei He
6. Frank K. Soong
7. Yanqing Liu
8. Bohan Li

Citations in the corpus

• AdaSpeech 4: Adaptive Text to Speech in Zero-Shot Scenarios (2022, methodology): “When synthesizing speech for new speakers, few-shot adaptive TTS [10, 11, 12, 13] is usually adopted by first training a source TTS model on a large multi-speaker dataset and then fine-tuning this model on a few speech data of target speakers.”
• Guided-TTS: A Diffusion Model for Text-to-Speech via Classifier Guidance (2021, methodology): “There have also been approaches using untranscribed speech to adapt the pre-trained multi-speaker TTS model for fewshot TTS synthesis (Jia et al., 2018; Yan et al., 2021).”
• A Survey on Neural Speech Synthesis (2021, background): “Tacotron 1/2 [72, 73] Ph→MelS NAR / Hybrid/Self-Att/CNN MelNet [361] Ch→MelS AR / RNN DeepVoice [8] Ch/Ph→MelS AR / CNN DeepVoice 2 [84] Ch/Ph→MelS AR / CNN DeepVoice 3 [264] Ch/Ph→MelS AR Seq2Seq CNN ParaNet [262] Ph→MelS NAR Seq2Seq CNN DCTTS [326] Ch→MelS AR Seq2Seq CNN SpeedySpeech [355] Ph→MelS”
• AdaSpeech: Adaptive Text to Speech for Custom Voice (2021, methodology): “We will also investigate the adaptation setting with untranscribed data (Yan et al., 2021) and further compress the model size (Luo et al., 2021) to support more custom voices.”
• RetrieverTTS: Modeling Decomposed Factors for Text-Based Speech Insertion (2022, methodology): “In the first line, methods [7, 8, 9, 10] achieve satisfactory voice quality and similarity by finetuning the pre-trained model on the reference voice before inference.”
• AdaSpeech 3: Adaptive Text to Speech for Spontaneous Style (2021, methodology): “For the acoustic condition modeling and conditional layer normalization as used in [17, 24], we integrate it in the phoneme encoder and mel decoder respectively, and do not explicitly show in Figure 1.”