Multi-task Sequence to Sequence Learning

By Minh-Thang Luong, Quoc V. Le, Ilya Sutskever, Oriol Vinyals, Lukasz Kaiser. 2015

836 citations recorded by the source corpus.

Sequence to sequence learning has recently emerged as a new paradigm in supervised learning. To date, most of its applications focused on only one task and not much work explored this framework for multiple tasks. This paper examines three multi-task learning (MTL) settings for sequence to sequence models: (a) the oneto-many setting - where the encoder is shared between several tasks such as machine translation and syntactic parsing, (b) the many-to-one setting - useful when only the decoder can be shared, as in the case of translation and image caption generation, and (c) the many-to-many setting - where multiple encoders and decoders are shared, which is the case with unsupervised objectives and translation. Our results show that training on a small amount of parsing and image caption data can improve the translation quality between English and German by up to 1.5 BLEU points over strong single-task baselines on the WMT benchmarks. Furthermore, we have established a new state-of-the-art result in constituent parsing with 93.0 F1. Lastly, we reveal interesting properties of the two unsupervised learning objectives, autoencoder and skip-thought, in the MTL context: autoencoder helps less in terms of perplexities but more on BLEU scores compared to skip-thought.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Attention is All you Need (2017, neutral): “(2014) [37] semi-supervised 92.1 Transformer (4 layers) semi-supervised 92.7 Luong et al. (2015) [23] multi-task 93.0 Dyer et al. (2016) [8] generative 93.3 increased the maximum output length to input length + 300 .”
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation (2016, background): “Though this can be addressed in principle by training a “copy model” to mimic a traditional alignment model [31], or by using the attention mechanism to copy rare words [37], these approaches are both unreliable at scale, since the quality of the alignments varies across languages, and the latent…”
Multi-Task Deep Neural Networks for Natural Language Understanding (2019, background): “Recently, there is a growing interest in applying MTL to representation learning using deep neural networks (DNNs) (Collobert et al., 2011; Liu et al., 2015; Luong et al., 2015; Xu et al., 2018) for two reasons.”
The Natural Language Decathlon: Multitask Learning as Question Answering (2018, methodology): “First, the LSTM produces an intermediate state ht using the previous answer word At−1self and recurrent context state [Luong et al., 2015b]: LSTM([(Aself )t−1 ; c̃t−1], ht−1) = ht ∈ R d (7) Context and Question Attention.”
CTRL: A Conditional Transformer Language Model for Controllable Generation (2019, background): “…to extend proposals of uniﬁed, multi-task systems for all of NLP (Collobert & Weston, 2008; Collobert et al., 2011), parsing and tagging (Hashimoto et al., 2016), multiple languages (Wu et al., 2016; Johnson et al., 2017), and multiple modalities (Luong et al., 2015; Kaiser et al., 2017).”
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, methodology): “Early multilingual NMT models utilized separate vo-cabularies for each language (Dong et al., 2015; Luong et al., 2015; Firat et al., 2016a); later ones used shared multilingual vocabularies (Sennrich et al., 2016; Ha et al., 2016c; Johnson et al., 2017) which are shared across languages.”
Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling (2018, background): “Luong et al. (2016) show promising results combining translation and parsing; Subramanian et al. (2018) benefit from multitask learning in sentence-to-vector encoding; and Bingel and Søgaard (2017) and Changpinyo et al. (2018) offer studies of when multitask learning is helpful for lower-level NLP…”
Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning (2018, background): “In contrast, Luong et al. (2015) alter the training ratios for each task based on the size of their respective training sets.”
Unsupervised Pretraining for Sequence to Sequence Learning (2016, background): “Various forms of transfer or multitask learning with seq2seq framework also have the flavors of our algorithm (Zoph et al., 2016; Luong et al., 2015; Firat et al., 2016).”
BAM! Born-Again Multi-Task Networks for Natural Language Understanding (2019, background): “Multi-task learning for neural networks in general (Caruana, 1997) and within NLP speciﬁcally (Col-lobert and Weston, 2008; Luong et al., 2016) has been widely studied.”
Finetuned Language Models Are Zero-Shot Learners (2021, background): “…tuning can be seen as a formulation of multitask learning (MTL), which is an established area within deep learning (Collobert et al., 2011; Luong et al., 2016; Ruder, 2017; Velay & Daniel, 2018; Clark et al., 2019b; Liu et al., 2019b, inter alia)—see Worsham & Kalita (2020) for a recent…”
A Knowledge-Grounded Neural Conversation Model (2017, methodology): “This yields the following reranking score: log P ( R | S, F ) + λ log P ( S | R ) + γ | R | λ and γ are free parameters, which we tune on our development N -best lists using MERT (Och, 2003) by optimizing B LEU (Papineni et al., 2002a).”
Perceiver IO: A General Architecture for Structured Inputs & Outputs (2021, background): “In language, training or evaluation on multiple tasks has also become common [15, 21, 50, 44, 63].”
Robust Speech Recognition via Large-Scale Weak Supervision (2022, neutral): “Multitask learning in the sequence-to-sequence framework (Sutskever et al., 2014) using multiple encoders and decoders was investigated in Luong et al. (2015).”
Google’s Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation (2016, methodology): “Despite training on WMT’14 data, which is somewhat smaller than WMT’15, we test our De→En model on newstest2015, similar to Luong et al., (2015b). The combination of newstest2012 and newstest2013 is used as the development set.”
Controllable Abstractive Summarization (2017, background): “, 2015b), parsing (Luong et al., 2015a) and image captioning (Vinyals et al.”