MAM: Masked Acoustic Modeling for End-to-End Speech-to-Text Translation

By Junkun Chen, Mingbo Ma, Renjie Zheng, Liang Huang. 2020

21 citations recorded by the source corpus.

End-to-end Speech-to-text Translation (E2E- ST), which directly translates source language speech to target language text, is widely useful in practice, but traditional cascaded approaches (ASR+MT) often suffer from error propagation in the pipeline. On the other hand, existing end-to-end solutions heavily depend on the source language transcriptions for pre-training or multi-task training with Automatic Speech Recognition (ASR). We instead propose a simple technique to learn a robust speech encoder in a self-supervised fashion only on the speech side, which can utilize speech data without transcription. This technique, termed Masked Acoustic Modeling (MAM), can also perform pre-training, for the first time, on any acoustic signals (including non-speech ones) without annotation. Compared with current state-of-the-art models on ST, our technique achieves +1.4 BLEU improvement without using transcriptions, and +1.2 BLEU using transcriptions. The pre-training of MAM with arbitrary acoustic signals also boosts the downstream speech-related tasks.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Mingbo Ma
2. Liang Huang
3. Junkun Chen
4. Renjie Zheng
5. P. Xu
6. Xiatian Zhu
7. David A. Clifton
8. Richard He Bai

Citations in the corpus

• A3T: Alignment-Aware Acoustic and Text Pretraining for Speech Synthesis and Editing (2022, positive): “On the other hand, another line of work, such as MAM (Chen et al., 2020) and FAT-MLM (Zheng et al.”
• Fused Acoustic and Text Encoding for Multimodal Bilingual Pretraining and Speech Translation (2021, positive): “• ST + MAM : ST trained with additional MAM loss (Chen et al., 2020) Table 7.”
• Multimodal Learning With Transformers: A Survey (2022, background): “, Masked Acoustic Modelling [180], [200], Masked Image Re-”