End-to-end Speech Translation via Cross-modal Progressive Training

By Rong Ye, Mingxuan Wang, Lei Li. 2021

81 citations recorded by the source corpus.

End-to-end speech translation models have become a new trend in research due to their potential of reducing error propagation. However, these models still suffer from the challenge of data scarcity. How to effectively use unlabeled or other parallel corpora from machine translation is promising but still an open problem. In this paper, we propose Cross Speech-Text Network (XSTNet), an end-to-end model for speech-to-text translation. XSTNet takes both speech and text as input and outputs both transcription and translation text. The model benefits from its three key design aspects: a self-supervised pre-trained sub-network as the audio encoder, a multi-task training objective to exploit additional parallel bilingual text, and a progressive training procedure. We evaluate the performance of XSTNet and baselines on the MuST-C En-X and LibriSpeech En-Fr datasets. In particular, XSTNet achieves state-of-the-art results on all language directions with an average BLEU of 28.8, outperforming the previous best method by 3.2 BLEU. Code, models, cases, and more detailed analysis are available at https://github.com/ReneeYe/XSTNet.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Shujie Liu
3. Yu Wu
4. Furu Wei
5. Long Zhou
6. Tom Ko
7. Yao Qian
8. Mingxuan Wang

Citations in the corpus

• SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing (2021, background): “Besides, a series of research work begins to investigate joint text and speech training (Han et al., 2021; Ye et al., 2021; Tang et al., 2021a; Zheng et al., 2021; Tang et al., 2021b), but they are mainly designed for speech to text tasks.”
• DUB: Discrete Unit Back-translation for Speech Translation (2023, background): “…2019b; Dong et al., 2021a,b; Zheng et al., 2021) and training techniques, such as multi-task learning (Indurthi et al., 2021; Tang et al., 2021; Ye et al., 2021), knowledge distillation (Liu et al., 2019; Dong et al., 2021b), and pre-training methods (Zheng et al., 2021; Zhang et al., 2022c),…”