Self-Training for End-to-End Speech Translation

By Juan Pino, Qiantong Xu, Xutai Ma, Mohammad Javad Dousti, Yun Tang. 2020

69 citations recorded by the source corpus.

One of the main challenges for end-to-end speech translation is data scarcity. We leverage pseudo-labels generated from unlabeled audio by a cascade and an end-to-end speech translation model. This provides 8.3 and 5.7 BLEU gains over a strong semi-supervised baseline on the MuST-C English-French and English-German datasets, reaching state-of-the art performance. The effect of the quality of the pseudo-labels is investigated. Our approach is shown to be more effective than simply pre-training the encoder on the speech recognition task. Finally, we demonstrate the effectiveness of self-training by directly generating pseudo-labels with an end-to-end model instead of a cascade model.

Subfields: Artificial Intelligence, Machine Translation, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Machine Translation

In Speech & Audio

In Signal Processing

Citations in the corpus

VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation (2021, background): “…et al., 2018; Schneider et al., 2019; Baevski et al., 2020; Conneau et al., 2020; Wu et al., 2020; Nguyen et al., 2020), with semi-supervised learning (self-training) (Kahn et al., 2020a; Pino et al., 2020; Zhang et al., 2020b; Xu et al., 2020) or a combination of both methods (Xu et al., 2020).”
Fairseq S2T: Fast Speech-to-Text Modeling with Fairseq (2020, background): “When supervised data is not abundant, self-supervised pre-training (Schneider et al., 2019; Wu et al., 2020) and semi-supervised training (Kahn et al., 2020; Pino et al., 2020) lowers the requirements on super-vision and improves model performance.”
Multilingual Speech Translation from Efficient Finetuning of Pretrained Models (2021, background): “…et al., 2019; Wang et al., 2020c,d; Indurthi et al., 2020; Di Gangi et al., 2019), pretraining different components of the model (B´erard et al., 2018; Bansal et al., 2019), transfer learning (Gaido et al., 2020; Liu et al., 2019) and generating synthetic data (Jia et al., 2018; Pino et al., 2020).”
End-to-end Speech Translation via Cross-modal Progressive Training (2021, methodology): “Techniques, such as pre-training [4, 5, 14, 3], multitask learning [15, 2], self-training [16] and meta-learning [1] have further improved the performance of end-to-end ST models.”
Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task (2021, result): “From Table 2, our “ST” baseline is comparable to the previously reported results except (Pino et al., 2020 (“JT” v.s. “JT-S-ASR”).”
Learning Shared Semantic Space for Speech-to-Text Translation (2021, unknown): “…training techniques such as pretraining (Weiss et al., 2017; B´erard et al., 2018; Bansal et al., 2019; Stoian et al., 2020; Wang et al., 2020a; Pino et al., 2020), multi-task training (Vydana et al., 2021; Le et al., 2020; Tang et al., 2021), meta-learning (Indurthi et al., 2020), and…”
DUB: Discrete Unit Back-translation for Speech Translation (2023, unknown): “…ST (In-aguma et al., 2020), E2E-ST-JT (Du et al., 2022), Speechformer (Papi et al., 2021), Cas-caded ST (Inaguma et al., 2020), MTL (Tang et al., 2021), Self-training (Pino et al., 2020), SpeechT5 (Ao et al., 2022), SpeechUT (Zhang et al., 2022c) and Revisit ST (Zhang et al., 2022a). et al., 2021).”