SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Training

By Ankur Bapna, Yu-An Chung, Na Wu, Anmol Gulati, Ye Jia, J. Clark, Melvin Johnson, Jason Riesa, Alexis Conneau, Yu Zhang. 2021

102 citations recorded by the source corpus.

Unsupervised pre-training is now the predominant approach for both text and speech understanding. Self-attention models pre-trained on large amounts of unannotated data have been hugely successful when fine-tuned on downstream tasks from a variety of domains and languages. This paper takes the universality of unsupervised language pre-training one step further, by unifying speech and text pre-training within a single model. We build a single encoder with the BERT objective on unlabeled text together with the w2v-BERT objective on unlabeled speech. To further align our model representations across modalities, we leverage alignment losses, specifically Translation Language Modeling (TLM) and Speech Text Matching (STM) that make use of supervised speech-text recognition data. We demonstrate that incorporating both speech and text data during pre-training can significantly improve downstream quality on CoVoST~2 speech translation, by around 1 BLEU compared to single-modality pre-trained models, while retaining close to SotA performance on LibriSpeech and SpeechStew ASR tasks. On four GLUE tasks and text-normalization, we observe evidence of capacity limitations and interference between the two modalities, leading to degraded performance compared to an equivalent text-only model, while still being competitive with BERT. Through extensive empirical analysis we also demonstrate the importance of the choice of objective function for speech pre-training, and the beneficial effect of adding additional supervised signals on the quality of the learned representations.

Subfields: Artificial Intelligence, Natural Language Processing, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Speech & Audio

Citations in the corpus

MAESTRO: Matched Speech Text Representations through Modality Matching (2022, background): “One approach uses multi-task training which trains a single model with different objectives for each modality [5, 10, 11], while another uses modality conversion where text is directly converted to speech [12] via text-to-speech (TTS) approaches or viceversa [13] prior to self-supervised training.”
A3T: Alignment-Aware Acoustic and Text Pretraining for Speech Synthesis and Editing (2022, neutral): “For example, one line of work, such as wav2vec 2.0 (Baevski et al., 2020), Hubert (Hsu et al., 2021) and SLAM (Bapna et al., 2021), learn discrete quantized speech units as latent representations.”
End-to-End Speech Recognition: A Survey (2023, unknown): “into a shared space [320], [321], [322], [323], [324], [325], [130], [326].”
Textually Pretrained Speech Language Models (2023, background): “Nonetheless, previous work showed that speech and text LMs can be trained jointly to improve either machine translation [Bapna et al., 2021, Cheng et al., 2022, Bapna et al., 2022], or transcription based speech related tasks [Ao et al., 2021, Chen et al., 2023].”
SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-training (2022, methodology): “One line of speech-text joint pre-training builds a shared encoder to learn speech and text representation jointly, such as SLAM (Bapna et al., 2021), which needs a random initialization of the de-coder parameter for fine-tuning an encoder-decoder model.”
VoxtLM: Unified Decoder-Only Models for Consolidating Speech Recognition, Synthesis and Speech, Text Continuation Tasks (2023, background): “Several studies [11,12,17] pro-pose to learn shared speech-text representations in a self-supervised manner.”
CoBERT: Self-Supervised Speech Representation Learning Through Code Representation Learning (2022, background): “Another line of research aims at utilizing unpaired text to benefit speech representation learning [11, 12, 13].”
Self-Supervised Speech Representation Learning: A Review (2022, methodology): “The SLAM approach [191] is an example where speech and text are first represented using two separate pre-trained encoders followed by a multi-modal encoder to build the joint”
SpeechLM: Enhanced Speech Pre-Training With Unpaired Textual Data (2022, methodology): “Moreover, SpeechLM-P Large model surprisingly outperforms previous work such as SLAM X-Large [6], even though the latter has a much bigger model size.”
Leveraging unsupervised and weakly-supervised data to improve direct speech-to-speech translation (2022, methodology): “To this end, we use speech-text joint representations learned from an mSLAM [20, 21] model.”