One-to-Many Multilingual End-to-End Speech Translation

By Mattia Antonino Di Gangi, Matteo Negri, Marco Turchi. 2019

53 citations recorded by the source corpus.

Nowadays, training end-to-end neural models for spoken language translation (SLT) still has to confront with extreme data scarcity conditions. The existing SLT parallel corpora are indeed orders of magnitude smaller than those available for the closely related tasks of automatic speech recognition (ASR) and machine translation (MT), which usually comprise tens of millions of instances. To cope with data paucity, in this paper we explore the effectiveness of transfer learning in end-to-end SLT by presenting a multilingual approach to the task. Multilingual solutions are widely studied in MT and usually rely on “target forcing”, in which multilingual parallel data are combined to train a single model by prepending to the input sequences a language token that specifies the target language. However, when tested in speech translation, our experiments show that MT-like target forcing, used as is, is not effective in discriminating among the target languages. Thus, we propose a variant that uses target-language embed-dings to shift the input representations in different portions of the space according to the language, so to better support the production of output in the desired target language. Our experiments on end-to-end SLT from English into six languages show important improvements when translating into similar languages, especially when these are supported by scarce data. Further improvements are obtained when using English ASR data as an additional language (up to +2.5 BLEU points).

Subfields: Artificial Intelligence, Machine Translation, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Machine Translation

In Speech & Audio

Citations in the corpus

Fairseq S2T: Fast Speech-to-Text Modeling with Fairseq (2020, methodology): “Compared with previous Transformer-based approaches (Di Gangi et al., 2019b; Inaguma et al., 2020), our bilingual models achieve comparative results to the state of the art without applying additional techniques such as speed perturbation and pre-trained decoder from MT.”
Multilingual Speech Translation from Efficient Finetuning of Pretrained Models (2021, background): “…on multi-task learning (Weiss et al., 2017; Anastasopoulos and Chiang, 2018; Bahar et al., 2019; Wang et al., 2020c,d; Indurthi et al., 2020; Di Gangi et al., 2019), pretraining different components of the model (B´erard et al., 2018; Bansal et al., 2019), transfer learning (Gaido et al.,…”
Self-Training for End-to-End Speech Translation (2020, neutral): “Data scarcity has been addressed in previous work with data augmentation [1, 2], multi-task training [3, 4], pre-training [5, 6] or multilingual speech translation [7, 8, 9].”
Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task (2021, methodology): “For reference, we also include the MT evaluation results from MUSTC (Gangi et al., 2019a) in the second row.”
ESPnet-ST: All-in-One Speech Translation Toolkit (2020, methodology): “Must-C corpus contains English speech extracted from TED talks, the corresponding transcription, and the target translations in 8 language directions (De, Pt, Fr, Es, Ro, Ru, Nl, and It) (Di Gangi et al., 2019a).”
Improving Cross-Lingual Transfer Learning for End-to-End Speech Recognition with Speech Translation (2020, background): “When targeting at ST only, transfer learning from ASR [15, 13] is helpful to warm-starting acoustic modeling (encoder) and enabling ST model training to focus more on learning language modeling and alignment (decoder).”
FINDINGS OF THE IWSLT 2020 EVALUATION CAMPAIGN (2020, methodology): “For English → Czech Translation: • All the data for English-German track by WMT 2019 25 News Translation Task, i.e.: – English-German parallel data, – German monolingual data, • MuST-C (Di Gangi et al., 2019a), release 1.0 contains English-German pair, • Speech-Translation TED corpus, the English-Ge”
A General Multi-Task Learning Framework to Leverage Text Data for Speech to Text Tasks (2020, result): “The results from baseline (row 4) are comparable to results in [29], though all baselines are trained from scratch with random initialization.”
Translatotron 2: Robust direct speech-to-speech translation (2021, background): “…recently, direct speech-to-text translation (ST) is rapidly emerging, and has outperformed the cascade of ASR and MT (Weiss et al., 2017; Jia et al., 2019a; Di Gangi et al., 2019; McCarthy et al., 2020; Wang et al., 2021), which makes the cascade of ST and TTS as S2ST feasible (Jia et al., 2019b).”
Large-Scale Self- and Semi-Supervised Learning for Speech Translation (2021, background): “Similarly for speech translation, there has been work on combating data scarcity, namely multitask learning [11, 12, 13], pretraining on ASR data [12, 14, 15, 16], data augmentation [17, 18, 19, 20], self-supervised pretraining [21, 22], selftraining [23] or multilingual speech translation [24, 25, ”
CoVoST 2: A Massively Multilingual Speech-to-Text Translation Corpus (2020, background): “MuST-C provides TED talks translations from English into 8 European languages, with data amounts ranging from 385 hours to 504 hours, thereby encouraging research into end-to-end ST (Berard et al., 2016) as well as one-to-many multilingual ST (Di Gangi et al., 2019).”
Translatotron 2: High-quality direct speech-to-speech translation with voice preservation (2021, background): “…direct speech-to-text translation (ST) is rapidly emerging, and has outperformed the cascade of ASR and MT (Weiss et al., 2017; Jia et al., 2019a; Di Gangi et al., 2019; McCarthy et al., 2020; Ansari et al., 2020; Wang et al., 2021b; Anastasopoulos et al., 2021), which makes the Proceedings of…”
Speech Translation and the End-to-End Promise: Taking Stock of Where We Are (2020, background): “…of the latter approach, so follow-up work concentrates on compiling ST corpora from available web sources (Godard et al., 2018; Kocabiyikoglu et al., 2018; Sanabria et al., 2018; di Gangi et al., 2019a; Boito et al., 2020; Beilharz et al., 2020; Iranzo-Sánchez et al., 2020; Wang et al., 2020a).”