SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing

By Junyi Ao, Rui Wang, Long Zhou, Shujie Liu, Shuo Ren, Yu Wu, Tom Ko, Qing Li, Yu Zhang, Zhihua Wei and 3 more. 2021

297 citations recorded by the source corpus.

Motivated by the success of T5 (Text-To-Text Transfer Transformer) in pre-trained natural language processing models, we propose a unified-modal SpeechT5 framework that explores the encoder-decoder pre-training for self-supervised speech/text representation learning. The SpeechT5 framework consists of a shared encoder-decoder network and six modal-specific (speech/text) pre/post-nets. After preprocessing the input speech/text through the pre-nets, the shared encoder-decoder network models the sequence-to-sequence transformation, and then the post-nets generate the output in the speech/text modality based on the output of the decoder. Leveraging large-scale unlabeled speech and text data, we pre-train SpeechT5 to learn a unified-modal representation, hoping to improve the modeling capability for both speech and text. To align the textual and speech information into this unified semantic space, we propose a cross-modal vector quantization approach that randomly mixes up speech/text states with latent units as the interface between encoder and decoder. Extensive evaluations show the superiority of the proposed SpeechT5 framework on a wide variety of spoken language processing tasks, including automatic speech recognition, speech synthesis, speech translation, voice conversion, speech enhancement, and speaker identification.

Subfields: Artificial Intelligence, Multimodal Learning, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Multimodal Learning

In Speech & Audio

Citations in the corpus

MAESTRO: Matched Speech Text Representations through Modality Matching (2022, background): “One approach uses multi-task training which trains a single model with different objectives for each modality [5, 10, 11], while another uses modality conversion where text is directly converted to speech [12] via text-to-speech (TTS) approaches or viceversa [13] prior to self-supervised training.”
VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translation (2023, positive): “Inspired by T5 (Raffel et al., 2020), SpeechT5 (Ao et al., 2021) adopts a universal encoder-decoder based sequence-to-sequence model for all spoken language tasks, e.g., speech recognition, speech translation, speech synthesis, etc.”
WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing (2021, background): “It is also a promising direction to jointly learn text and speech representation in a self-supervised pre-training framework [88], as the huge amount of text data might increase the capability of speech content modeling.”
Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentation (2022, background): “SpeechT5 [25] proposes end-to-end pre-training, where the model learns to reconstruct the log Mel-filterbank of the masked regions of the input speech, to improve performance of speechto-speech tasks such as voice conversion and speech enhancement, while it was not evaluated in the context of direct”
Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervision (2023, methodology): “2 Overall, our approach to building TTS using massive self-supervised pretraining and backtranslation of discrete speech representations considerably differs from how existing TTS systems are implemented (Shen et al., 2018; Kong et al., 2020; Ren et al., 2020; Kim et al., 2021; Ao et al., 2022; Wang”
data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language (2022, background): “Work in selfsupervised learning for speech includes autoregressive models (van den Oord et al., 2018; Schneider et al., 2019; Baevski et al., 2020a; Chung et al., 2019) as well as bidirectional models (Baevski et al., 2020b; Hsu et al., 2021; Ao et al., 2021; Chen et al., 2021a).”
Qwen2-Audio Technical Report (2024, background): “To reduce the gap between pre-training and post-training stages, we simplify the Previous Top-tiers Qwen-Audio Qwen2-Audio Figure 1: Performance of Qwen2-Audio, Qwen-Audio and previous top-tiers from LALMs such as SpeechT5 (Ao et al., 2021), SpeechNet (Chen et al., 2021), SpeechLLaMA (Wu et al., 202”
End-to-End Speech Recognition: A Survey (2023, unknown): “into a shared space [320], [321], [322], [323], [324], [325], [130], [326].”
VoxtLM: Unified Decoder-Only Models for Consolidating Speech Recognition, Synthesis and Speech, Text Continuation Tasks (2023, methodology): “Several studies [11,12,17] pro-pose to learn shared speech-text representations in a self-supervised manner.”
DUB: Discrete Unit Back-translation for Speech Translation (2023, unknown): “…ST (In-aguma et al., 2020), E2E-ST-JT (Du et al., 2022), Speechformer (Papi et al., 2021), Cas-caded ST (Inaguma et al., 2020), MTL (Tang et al., 2021), Self-training (Pino et al., 2020), SpeechT5 (Ao et al., 2022), SpeechUT (Zhang et al., 2022c) and Revisit ST (Zhang et al., 2022a). et al., 2021).”
LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPT (2023, background): “The second group, such as SpeechT5 (Ao et al., 2022) and SpeechNet (Chen et al., 2021b), can perform various speech tasks with a single model, but they need to employ modal-specific pre-nets and post-nets to handle different input/output modalities.”
SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-training (2022, methodology): “Specifically, we generate the units from speech for a small paired dataset using the S2U generator, and then remove the repetitive units of adjacent frames to get reduced units (Ao et al., 2022b).”
Generative Pre-training for Speech with Flow Matching (2023, background): “Besides self-supervised learning, pre-training have also been studied in the context of semi-supervised TTS (Chung et al., 2019) or speech-text alignment (Ao et al., 2021), but these works focused on non-generative models.”
textless-lib: a Library for Textless Spoken Language Processing (2022, methodology): “This area of research has been made possible by converging progress in self-supervised speech representation learning (Schneider et al., 2019; Baevski et al., 2020; Oord et al., 2018; Hsu et al., 2021; Chorowski et al., 2021; Chen et al., 2021; Chung et al., 2021; Wang et al., 2021; Ao et al., 2021)”
Textually Pretrained Speech Language Models (2023, background): “Nonetheless, previous work showed that speech and text LMs can be trained jointly to improve either machine translation [Bapna et al., 2021, Cheng et al., 2022, Bapna et al., 2022], or transcription based speech related tasks [Ao et al., 2021, Chen et al., 2023].”
CoBERT: Self-Supervised Speech Representation Learning Through Code Representation Learning (2022, background): “Another line of research aims at utilizing unpaired text to benefit speech representation learning [11, 12, 13].”