MAESTRO: Matched Speech Text Representations through Modality Matching

By Zhehuai Chen, Yu Zhang, Andrew Rosenberg, Bhuvana Ramabhadran, Pedro J. Moreno Mengibar, Ankur Bapna, Heiga Zen. 2022

125 citations recorded by the source corpus.

We present Maestro, a self-supervised training method to unify representations learnt from speech and text modalities. Self-supervised learning from speech signals aims to learn the latent structure inherent in the signal, while self-supervised learning from text attempts to capture lexical information. Learning aligned representations from unpaired speech and text sequences is a challenging task. Previous work either implicitly enforced the representations learnt from these two modalities to be aligned in the latent space through multitasking and parameter sharing or explicitly through conversion of modalities via speech synthesis. While the former suffers from interference between the two modalities, the latter introduces additional complexity. In this paper, we propose Maestro, a novel algorithm to learn unified representations from both these modalities simultaneously that can transfer to diverse downstream tasks such as Automated Speech Recognition (ASR) and Speech Translation (ST). Maestro learns unified representations through sequence alignment, duration prediction and matching embeddings in the learned space through an aligned masked-language model loss. We establish a new state-of-the-art (SOTA) on VoxPopuli multilingual ASR with a 8% relative reduction in Word Error Rate (WER), multidomain SpeechStew ASR (3.7% relative) and 21 languages to English multilingual ST on CoVoST 2 with an improvement of 2.8 BLEU averaged over 21 languages.

Subfields: Multimodal Learning, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Multimodal Learning

In Speech & Audio

Citations in the corpus

Robust Speech Recognition via Large-Scale Weak Supervision (2022, methodology): “We compare with Maestro, mSLAM, and XLS-R, the highest-performing prior work.”
Scaling Speech Technology to 1, 000+ Languages (2023, background): “VoxPopuli [Wang et al., 2021] 1 15.3 Maestro [Chen et al., 2022] 14 8.1 RNN-T 1B [Li et al., 2021] 15 7.9 Whisper [Radford et al., 2022] 99 13.6 ∗ 7.3* ML-IO [Tjandra et al., et al., 2022].”
Building Machine Translation Systems for the Next Thousand Languages (2022, background): “Being able to train models that can learn from and represent speech and text jointly (Zheng et al., 2021; Bapna et al., 2022; Chen et al., 2022; Bai et al., 2022; Tang et al., 2022) is essential to alleviate data sparsity and building more robust language technologies for such languages.”
SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-training (2022, methodology): “While, w2v-Conformer (Zhang et al., 2020), w2v-BERT (Chung et al., 2021a), SLAM (Bapna et al., 2021), and Maestro (Chen et al., 2022) use much larger models with Conformer blocks and/or more speech/text data, which are beyond fair comparison.”
Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages (2023, methodology): “The upsampler used in this work is a learned duration based upsampling model [13], though a fixed or random repetition upsampler can also be used for text-injection [47, 53].”
End-to-End Speech Recognition: A Survey (2023, unknown): “into a shared space [320], [321], [322], [323], [324], [325], [130], [326].”
SpeechLM: Enhanced Speech Pre-Training With Unpaired Textual Data (2022, methodology): “MAESTRO [8] also uses paired speech-text data in RNN-T framework to encourage the alignment of the intermediate representations at a more ﬁne-grained level.”
VoxtLM: Unified Decoder-Only Models for Consolidating Speech Recognition, Synthesis and Speech, Text Continuation Tasks (2023, methodology): “Several studies [11,12,17] pro-pose to learn shared speech-text representations in a self-supervised manner.”