Adaptive Knowledge Sharing in Multi-Task Learning: Improving Low-Resource Neural Machine Translation

By Poorya Zaremoodi, Wray L. Buntine, Gholamreza Haffari. 2018

61 citations recorded by the source corpus.

Neural Machine Translation (NMT) is notorious for its need for large amounts of bilingual data. An effective approach to compensate for this requirement is Multi-Task Learning (MTL) to leverage different linguistic resources as a source of inductive bias. Current MTL architectures are based on the Seq2Seq transduction, and (partially) share different components of the models among the tasks. However, this MTL approach often suffers from task interference and is not able to fully capture commonalities among subsets of tasks. We address this issue by extending the recurrent units with multiple “blocks” along with a trainable “routing network”. The routing network enables adaptive collaboration by dynamic sharing of blocks conditioned on the task at hand, input, and model state. Empirical evaluation of two low-resource translation tasks, English to Vietnamese and Farsi, show +1 BLEU score improvements compared to strong baselines.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Machine Translation, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Machine Translation

In Meta & Continual Learning

Citations in the corpus

Massively Multilingual Neural Machine Translation (2019, neutral): “Zaremoodi et al. (2018) utilize recurrent units with multiple blocks together with a trainable routing network.”
ERNIE: Enhanced Language Representation with Informative Entities (2019, background): “…next layer. signiﬁcantly enhance original models, such as reading comprehension (Mihaylov and Frank, 2018; Zhong et al., 2018), machine translation (Zaremoodi et al., 2018), natural language inference (Chen et al., 2018), knowledge acquisition (Han et al., 2018a), and dialog systems (Madotto et…”
KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation (2019, background): “…not only many works integrating knowledge embed-dings into NLP models to improve the performance of NLP applications such as machine translation (Zaremoodi et al., 2018), reading comprehension (Mihaylov and Frank, 2018; Zhong et al., 2019) and dialogue system (Madotto et al., 2018), but also…”
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models (2018, unknown): “This kind of parameter sharing approach among multiple tasks was further extended in Zaremoodi and Haffari (2018); Zaremoodi et al. (2018) in which they also include semantic and syntactic parsing tasks and control the relative sharing of various parameters among the tasks to obtain accuracy gains…”
Recursive Routing Networks: Learning to Compose Modules for Language Understanding (2019, background): “Zaremoodi et al. (2018) introduced a soft version of routing that falls within a larger class of Mixtures of Experts (MoE) models (Jacobs et al., 1991).”
HyperGrid Transformers: Towards A Single Model for Multiple Tasks (2021, background): “(Zaremoodi et al., 2018) proposed Adaptive Knowledge Sharing 3 for low-resource neural machine translation.”
A Survey of Multilingual Neural Machine Translation (2019, background): “[164] proposed a routing network to dynamically control parameter sharing where the parts to be shared depend on the parallel corpora used for training.”
Balancing Training for Multilingual Neural Machine Translation (2020, background): “(Zaremoodi et al., 2018; Wang et al., 2018, 2019a) propose improvements to the model architecture to improve multilingual performance, while MultiDDS is a model-agnostic and optimizes multilingual data usage.”
HyperGrid: Efficient Multi-Task Transformers with Grid-wise Decomposable Hyper Projections (2020, background): “[Zaremoodi et al., 2018] proposed Adaptive Knowledge Sharing1 for low-resource neural machine translation.”
Neural Machine Translation for Low-resource Languages: A Survey (2021, background): “1Million, respectively [99, 109, 129, 132, 183]; however, these are not absolute values for the size of the corpora.”