Fused Acoustic and Text Encoding for Multimodal Bilingual Pretraining and Speech Translation

By Renjie Zheng, Junkun Chen, Mingbo Ma, Liang Huang. 2021

76 citations recorded by the source corpus.

Recently text and speech representation learning has successfully improved many language related tasks. However, all existing methods only learn from one input modality, while a unified acoustic and text representation is desired by many speech-related tasks such as speech translation. We propose a Fused Acoustic and Text Masked Language Model (FAT-MLM) which jointly learns a unified representation for both acoustic and text in-put. Within this cross modal representation learning framework, we further present an end-to-end model for Fused Acoustic and Text Speech Translation (FAT-ST). Experiments on three translation directions show that our proposed speech translation models fine-tuned from FAT-MLM substantially improve translation quality (+5.90 BLEU).

Subfields: Artificial Intelligence, Multimodal Learning, Machine Translation, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Multimodal Learning

In Machine Translation

In Speech & Audio

Citations in the corpus

MAESTRO: Matched Speech Text Representations through Modality Matching (2022, methodology): “by utilizing additional supervised data, joint unsupervised and supervised training on transcribed speech [25] or paired Masked Language Modeling (MLM) objectives on Machine Translation (MT) [26] or Speech Translation (ST) data [27].”
mSLAM: Massively multilingual joint pre-training for speech and text (2022, methodology): “…with SpanBERT (Joshi et al., 2020) and w2v-BERT (Chung et al., 2021) self-supervised losses that leverage unlabeled monomodal data, as well as a TLM loss (Conneau & Lample, 2019; Zheng et al., 2021) and a speech-text matching loss (Li et al., 2021) that both use supervised speech recognition data.”
A3T: Alignment-Aware Acoustic and Text Pretraining for Speech Synthesis and Editing (2022, neutral): “…tasks, such as speech recognition, speech classiﬁcation, and speech translation (Baevski et al., 2020; Chen et al., 2020; Liu et al., 2020; Zheng et al., 2021; Hsu et al., 2021) However, all these efforts can only support speech under-* standing tasks which take speech as input, but for…”
SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing (2021, background): “Besides, a series of research work begins to investigate joint text and speech training (Han et al., 2021; Ye et al., 2021; Tang et al., 2021a; Zheng et al., 2021; Tang et al., 2021b), but they are mainly designed for speech to text tasks.”
SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Training (2021, methodology): “…SpanBERT (Joshi et al., 2020) on unlabeled text, w2v-BERT (Chung et al., 2021) on unlabeled speech, Translation Language Modeling (Conneau & Lample, 2019; Zheng et al., 2021) on paired speech and text data, and Speech-Text Matching (Li et al., 2021) on paired and non-paired speech and text data.”
Multimodal Learning With Transformers: A Survey (2022, background): “For instance, Masked Language Modelling (MLM) in the text domain has been applied to audio and image, e.g., Masked Acoustic Modelling [180], [200], Masked Image Region Prediction [190], while both Sentence Ordering Modelling (SOM) [201] in text domain and Frame Ordering Modelling (FOM) [116] in vide”
DUB: Discrete Unit Back-translation for Speech Translation (2023, methodology): “, 2021b), and pretraining methods (Zheng et al., 2021; Zhang et al., 2022c), were developed to improve end-to-end performance.”
Building Machine Translation Systems for the Next Thousand Languages (2022, background): “Being able to train models that can learn from and represent speech and text jointly (Zheng et al., 2021; Bapna et al., 2022; Chen et al., 2022; Bai et al., 2022; Tang et al., 2022) is essential to alleviate data sparsity and building more robust language technologies for such languages.”
End-to-end Speech Translation via Cross-modal Progressive Training (2021, methodology): “Very recently, [6] proposed a Fused Acoustic and Text Masked Language Model to pre-train and improved ST by fine-tuning.”