Tied Multitask Learning for Neural Speech Translation
By Antonios Anastasopoulos, David Chiang. 2018
180 citations recorded by the source corpus.
We explore multitask models for neural translation of speech, augmenting them in order to reflect two intuitive notions. First, we introduce a model where the second task decoder receives information from the decoder of the first task, since higher-level intermediate representations should provide useful information. Second, we apply regularization that encourages transitivity and invertibility. We show that the application of these notions on jointly trained models improves performance on the tasks of low-resource speech transcription and translation. It also leads to better performance when using attention information for word discovery over unsegmented input.
Subfields: Artificial Intelligence, Machine Translation, Speech & Audio
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Graham Neubig
- Luke S. Zettlemoyer
- Mohit Bansal
- Kyunghyun Cho
- Felix Stahlberg
- Yonatan Belinkov
- Yoshua Bengio
- Jianfeng Gao
In Machine Translation
In Speech & Audio
Citations in the corpus
- Direct speech-to-speech translation with a sequence-to-sequence model (2019, background): “Other future work includes utilizing weakly supervision to scale up training with synthetic data [21] or multitask learning [19,20], and transferring prosody and other acoustic factors from the source speech to the translated speech following [45–47].”
- Multilingual Speech Translation from Efficient Finetuning of Pretrained Models (2021, methodology): “Multi-task learning has been shown as an effective approach to improve the performance of the speech translation task using other related tasks, such as MT and ASR (Weiss et al., 2017; Anastasopoulos and Chiang, 2018; Bahar et al., 2019; Tang et al., 2021a,b).”
- Enhancing Transformer for End-to-end Speech-to-Text Translation (2019, background): “For this reason, end-to-end solutions have been recently proposed (B´erard et al., 2016; Weiss et al., 2017; Anastasopoulos and Chiang, 2018; Liu et al., 2018; Di Gangi et al., 2018) but, in terms of performance, they are still far behind the pipeline approach.”
- Learning to Jointly Translate and Predict Dropped Pronouns with a Shared Reconstruction Mechanism (2018, result): “Our work is most similar to the work of Zoph and Knight (2016) and Anastasopoulos and Chiang (2018), which share a decoder and two separate attention models to read from two different sources.”
- Dynamic Task Prioritization for Multitask Learning (2018, methodology): “Second, at some point in the network architecture, hard sharing methods use a single shared representation which is then fed into multiple task submodules [39, 61, 53, 62, 8, 57].”
- MuST-C: a Multilingual Speech Translation Corpus (2019, background): “2 Besides the corpora reported in (Weiss et al., 2017; Anastasopoulos and Chiang, 2018; B´erard et al., 2018), iii) encoder/decoder pre-training (Bansal et al., 2018; B´erard et al., 2018), iv) synthesized speech data (B´erard et al., 2016), or v) machine-translated target text data (B´erard et…”
- Improving Cross-Lingual Transfer Learning for End-to-End Speech Recognition with Speech Translation (2020, background): “And very recent works [12, 13, 14] have demonstrated the possibility of combining the two related tasks, ASR and ST, under the same encoderdecoder architecture to achieve better performance.”
- Analyzing ASR Pretraining for Low-Resource Speech-to-Text Translation (2019, background): “Several of these use transcribed source language audio and/or translated source language text in a multitask learning scenario [4, 6, 9] or to pre-train parts of the model before fine-tuning on the end-to-end AST task [4].”
- Speech Translation and the End-to-End Promise: Taking Stock of Where We Are (2020, background): “(2017); Anastasopoulos and Chiang (2018); Wang et al.”
- Synchronous Speech Recognition and Speech-to-Text Translation with Interactive Decoding (2019, methodology): “Anastasopoulos and Chiang (2018a; 2018b) employed a triangle model on low-resource speech translation. Sperber et al. (2019) further applied an attention-passing mechanism which can integrate auxiliary data and improve model robustness.”
- ESPnet-ST: All-in-One Speech Translation Toolkit (2020, methodology): “, 2014), source speech and target text translation (without the intermediate source text transcript) might be easier to obtain, necessitating the adoption of E2E-ST models (Anastasopoulos and Chiang, 2018).”
- One-to-Many Multilingual End-to-End Speech Translation (2019, background): “It is widely demonstrated in literature that exploiting ASR data is useful for improving the performance of direct SLT systems, with the main approaches being transfer learning [16, 15] and multitask learning [14, 17].”
- Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task (2021, background): “Anastasopoulos and Chiang (2018) further extend it to a triangle structure by concatenating ASR and ST models.”
- Exploiting Deep Representations for Neural Machine Translation (2018, background): “Anastasopoulos and Chiang (2018) show that higher level layers are more representative than lower level layers.”