mSLAM: Massively multilingual joint pre-training for speech and text

By Ankur Bapna, Colin Cherry, Yu Zhang, Ye Jia, Melvin Johnson, Yong Cheng, Simran Khanuja, Jason Riesa, Alexis Conneau. 2022

133 citations recorded by the source corpus.

We present mSLAM, a multilingual Speech and LAnguage Model that learns cross-lingual cross-modal representations of speech and text by pre-training jointly on large amounts of unlabeled speech and text in multiple languages. mSLAM combines w2v-BERT pre-training on speech with SpanBERT pre-training on character-level text, along with Connectionist Temporal Classification (CTC) losses on paired speech and transcript data, to learn a single model capable of learning from and representing both speech and text signals in a shared representation space. We evaluate mSLAM on several downstream speech understanding tasks and find that joint pre-training with text improves quality on speech translation, speech intent classification and speech language-ID while being competitive on multilingual ASR, when compared against speech-only pre-training. Our speech translation model demonstrates zero-shot text translation without seeing any text translation data, providing evidence for cross-modal alignment of representations. mSLAM also benefits from multi-modal fine-tuning, further improving the quality of speech translation by directly leveraging text translation data during the fine-tuning process. Our empirical analysis highlights several opportunities and challenges arising from large-scale multimodal pre-training, suggesting directions for future research.

Subfields: Artificial Intelligence, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, methodology): “First, we transcribe the audio from all 3https://github.com/rmokady/CLIP prefix caption 4Note that due to the non-zero temperature used for sampling from the generative language model, results from this approach are stochastic, but comparable results are producible. videos with speech-to-text ALMs (”
Robust Speech Recognition via Large-Scale Weak Supervision (2022, methodology): “, 2020c) and mSLAM (Bapna et al., 2022) studied joint training over both text and speech language tasks, demonstrating transfer between them.”
MAESTRO: Matched Speech Text Representations through Modality Matching (2022, methodology): “We concatenate paired MT text sequences from CoVoST, WMT and TED datasets [11] and use them as unspoken text for pretraining.”
Visual Programming: Compositional visual reasoning without training (2022, background): “Socratic models (SMs) [33], compose pretrained models from different modalities such as language (BERT [7], GPT-2 [21]), vision-language (CLIP [20]), and audio-language (mSLAM [3]), to perform a number of zero-shot tasks, including image captioning, videoto-text retrieval, and robot planning.”
AudioPaLM: A Large Language Model That Can Speak and Listen (2023, neutral): “, 2018] text-to-text translation datasets as described in [Bapna et al., 2022].”
SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-training (2022, methodology): “When fine-tuning an encoder-decoder model, a randomly initialized de-coder needs to be superimposed on the encoder for speech-to-text tasks (Bapna et al., 2021, 2022).”
Textually Pretrained Speech Language Models (2023, background): “Nonetheless, previous work showed that speech and text LMs can be trained jointly to improve either machine translation [Bapna et al., 2021, Cheng et al., 2022, Bapna et al., 2022], or transcription based speech related tasks [Ao et al., 2021, Chen et al., 2023].”
Scaling Speech Technology to 1, 000+ Languages (2023, background): “Multilingual machine translation models have also been scaled to 200 languages [NLLB_Team et al., 2022] and even 1,000 languages [Bapna et al., 2022a].”
XTREME-S: Evaluating Cross-lingual Speech Representations (2022, methodology): “With more than 400k hours of unlabeled speech, VoxPopuli is also used as a public pre-training corpus [5, 14].”
Joint Pre-Training with Speech and Bilingual Text for Direct Speech to Speech Translation (2022, unknown): “For example, the paper in [9] explores pre-training the encoder with mSLAM objective [10], and pre-training the decoder of Translatoron 2 [11] with MT task to generate phonemes.”
Leveraging unsupervised and weakly-supervised data to improve direct speech-to-speech translation (2022, methodology): “4k hours of paired speech-transcript data spanning 32 languages, as described in [21].”
Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages (2023, methodology): “Following [20], we use CoVoST 2 [18] to benchmark multilingual speech translation.”
Building Machine Translation Systems for the Next Thousand Languages (2022, background): “Being able to train models that can learn from and represent speech and text jointly (Zheng et al., 2021; Bapna et al., 2022; Chen et al., 2022; Bai et al., 2022; Tang et al., 2022) is essential to alleviate data sparsity and building more robust language technologies for such languages.”
Textless Direct Speech-to-Speech Translation with Discrete Speech Representation (2022, methodology): “Instead of training a speech encoder from scratch, we initialize our encoder from a pre-trained multilingual w2v-BERT model [24] as described in [26]. w2v-BERT is a self-supervised model which combines both contrastive learning and masked language model (MLM) [24].”