A3T: Alignment-Aware Acoustic and Text Pretraining for Speech Synthesis and Editing

By Richard He Bai, Renjie Zheng, Junkun Chen, Xintong Li, Mingbo Ma, Liang Huang. 2022

66 citations recorded by the source corpus.

Recently, speech representation learning has improved many speech-related tasks such as speech recognition, speech classification, and speech-to-text translation. However, all the above tasks are in the direction of speech understanding, but for the inverse direction, speech synthesis, the potential of representation learning is yet to be realized, due to the challenging nature of generating high-quality speech. To address this problem, we propose our framework, Alignment-Aware Acoustic-Text Pretraining (A$^3$T), which reconstructs masked acoustic signals with text input and acoustic-text alignment during training. In this way, the pretrained model can generate high quality reconstructed spectrogram, which can be applied to the speech editing and unseen speaker TTS directly. Experiments show A$^3$T outperforms SOTA models on speech editing, and improves multi-speaker speech synthesis without the external speaker verification model.

Subfields: Multimodal Learning, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Multimodal Learning

1. Vimal Manohar
2. Zhuo Chen
3. Qiang Yang
4. Jinyu Li
5. Bowen Shi
6. Yossi Adi
7. Wei-Ning Hsu
8. Zhou Zhao

In Speech & Audio

1. Jinyu Li
2. Yi Ren
3. Zhou Zhao
4. Wei-Ning Hsu
5. Rongjie Huang
6. Jinglin Liu
7. K. Yu
8. Yossi Adi

Citations in the corpus

• SpeechX: Neural Codec Language Model as a Versatile Speech Transformer (2023, methodology): “For instance, existing speech editing models [20], [21] are restricted to handling clean signals only, lacking the ability to modify spoken content while preserving background sounds.”
• Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale (2023, methodology): “It aims to predict speech given context [36, 4] and optionally text guidance [3, 5, 61].”
• Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias (2023, methodology): “We compare the quality of generated audio samples of our Mega-TTS with SOTA speech editing baselines, including 1) EditSpeech [52]; 2) A(3)T [3].”
• DiffVoice: Text-to-Speech with Latent Diffusion (2023, methodology): “We further demonstrate that zero-shot adaptation can be solved by DiffVoice with state-of-the-art performance, by casting it as a speech continuation [20], or insertion problem [21].”
• F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching (2024, methodology): “Following Voicebox [14], we trained our model on the text-guided speech-infilling task.”
• Building Machine Translation Systems for the Next Thousand Languages (2022, background): “Being able to train models that can learn from and represent speech and text jointly (Zheng et al., 2021; Bapna et al., 2022; Chen et al., 2022; Bai et al., 2022; Tang et al., 2022) is essential to alleviate data sparsity and building more robust language technologies for such languages.”
• Generative Pre-training for Speech with Flow Matching (2023, methodology): “Using WavLM-TDCNN speaker embedding model Chen et al. (2022), speaker similarity is measured by the similarity between the embedding of generated speech and that of the conditioning audio.”