Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task

By Yun Tang, Juan Pino, Xian Li, Changhan Wang, Dmitriy Genzel. 2021

96 citations recorded by the source corpus.

Pretraining and multitask learning are widely used to improve the speech translation performance. In this study, we are interested in training a speech translation model along with an auxiliary text translation task. We conduct a detailed analysis to understand the impact of the auxiliary task on the primary task within the multitask learning framework. Our analysis confirms that multitask learning tends to generate similar decoder representations from different modalities and preserve more information from the pretrained text translation modules. We observe minimal negative transfer effect between the two tasks and sharing more parameters is helpful to transfer knowledge from the text task to the speech task. The analysis also reveals that the modality representation difference at the top decoder layers is still not negligible, and those layers are critical for the translation quality. Inspired by these findings, we propose three methods to improve translation quality. First, a parameter sharing and initialization strategy is proposed to enhance information sharing between the tasks. Second, a novel attention-based regularization is proposed for the encoders and pulls the representations from different modalities closer. Third, an online knowledge distillation is proposed to enhance the knowledge transfer from the text to the speech task. Our experiments show that the proposed approach improves translation performance by more than 2 BLEU over a strong baseline and achieves state-of-the-art results on the MuST-C English-German, English-French and English-Spanish language pairs.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Furu Wei
2. Juan Pino
3. Jinyu Li
4. Changhan Wang
5. Shujie Liu
6. Michael Auli
7. Alexei Baevski
8. Alexis Conneau

Citations in the corpus

• SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing (2021, background): “Besides, a series of research work begins to investigate joint text and speech training (Han et al., 2021; Ye et al., 2021; Tang et al., 2021a; Zheng et al., 2021; Tang et al., 2021b), but they are mainly designed for speech to text tasks.”
• Multilingual Speech Translation from Efficient Finetuning of Pretrained Models (2021, methodology): “Parameters in those 12 layers are shared between the two encoders during joint training (Tang et al., 2021b).”
• DUB: Discrete Unit Back-translation for Speech Translation (2023, background): “…2021), Espnet ST (In-aguma et al., 2020), E2E-ST-JT (Du et al., 2022), Speechformer (Papi et al., 2021), Cas-caded ST (Inaguma et al., 2020), MTL (Tang et al., 2021), Self-training (Pino et al., 2020), SpeechT5 (Ao et al., 2022), SpeechUT (Zhang et al., 2022c) and Revisit ST (Zhang et al.,…”
• A Metaverse: Taxonomy, Components, Applications, and Open Challenges (2022, unknown): “Tang et al. [294] proposed a parameter sharing and initialization strategy to enhance information sharing between tasks.”