Contextual Parameter Generation for Universal Neural Machine Translation

By Emmanouil Antonios Platanios, Mrinmaya Sachan, Graham Neubig, Tom Michael Mitchell. 2018

152 citations recorded by the source corpus.

We propose a simple modification to existing neural machine translation (NMT) models that enables using a single universal model to translate between multiple languages while allowing for language specific parameterization, and that can also be used for domain adaptation. Our approach requires no changes to the model architecture of a standard NMT system, but instead introduces a new component, the contextual parameter generator (CPG), that generates the parameters of the system (e.g., weights in a neural network). This parameter generator accepts source and target language embeddings as input, and generates the parameters for the encoder and the decoder, respectively. The rest of the model remains unchanged and is shared across all languages. We show how this simple modification enables the system to use monolingual data for training and also perform zero-shot translation. We further show it is able to surpass state-of-the-art performance for both the IWSLT-15 and IWSLT-17 datasets and that the learned language embeddings are able to uncover interesting relationships between languages.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Massively Multilingual Neural Machine Translation (2019, background): “Platanios et al. (2018) propose to share the entire network, while using a contextual parameter generator that learns to generate the parameters of the system given the desired source and target languages.”
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “More recently, with the increased number of languages (Qi et al., 2018; Aharoni et al., 2019), there has been work along the lines of applying soft sharing for MT, in the form of a shared set of meta-learned parameters used to generate task level parameters (Platanios et al., 2018).”
A Survey on Neural Speech Synthesis (2021, background): “AdaSpeech [39] proposes conditional layer normalization to generate the scale and bias parameters in layer normalization from the speaker embeddings based on contextual parameter generation [266] and only fine-tune the parameters related to the conditional layer normalization and speaker embeddings ”
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation (2020, neutral): “erators (Platanios et al., 2018), decoupling multilingual word encodings (Wang et al.”
Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks (2021, methodology): “In contemporaneous work, ¨Ust¨un et al. (2020) proposed a multilingual dependency parsing method based on adapters and contextual parameter generator networks (Platanios et al., 2018) where they generate adapter parameters conditioned on trained input language embeddings.”
Improved Zero-shot Neural Machine Translation via Ignoring Spurious Correlations (2019, methodology): “Platanios et al. (2018) introduced the contextual parameter generator, which generated the parameters of the system and performed zero-shot translation.”
Multilingual Neural Machine Translation with Language Clustering (2019, background): “Multilingual NMT NMT has recently been extended from the translation of a language pair to multilingual translation (Dong et al., 2015; Luong et al., 2015a; Firat et al., 2016; Lu et al., 2018; Johnson et al., 2017; He et al., 2019; Ha et al., 2016; Platanios et al., 2018; Tan et al., 2019).”
Meta-Learning Extractors for Music Source Separation (2020, background): “The definition is further constrained with M ′ < M so that the mapping by Pk extracts the most relevant information from ei which is then transformed with Wk to yield all the parameters of the k-th layer (similar to [2]).”
CondConv: Conditionally Parameterized Convolutions for Efficient Inference (2019, unknown): “[32] generate”
Share or Not? Learning to Schedule Language-Specific Capacity for Multilingual Translation (2021, methodology): “The incapability of full-space exploration motivates methods relying on heuristics (Sachan & Neubig, 2018), that lack flexibility when more languages are covered, or meta-learning (Platanios et al., 2018), that are often hard to scale.”
Neural Machine Translation for Low-resource Languages: A Survey (2021, background): “However, using word-level input results in large vocabulary sizes that are difficult to scale [129].”
Review of end-to-end speech synthesis technology based on deep learning (2021, methodology): “They used a fully convolutional encoder from DCTTS, whose parameters are generated using a separate contextual parameter generator network [163] conditioned on language embedding, thus realizing cross-lingual knowledge-sharing.”
A Survey of Deep Active Learning (2020, unknown): “NLP Machine translation [164, 248] OPUS [220], UNPC [261], IWSLT, WMT [163] Ind-En, Ch-En, En-Vi, Fr-En, En-De, etc.”
Consistency by Agreement in Zero-Shot Neural Machine Translation (2019, background): “…model and prepend special symbols to the source text to indicate the target language, which has later been extended to other text preprocessing approaches (Ha et al., 2017) as well as languageconditional parameter generation for encoders and decoders of a single model (Platanios et al., 2018).”
UDapter: Language Adaptation for Truly Universal Dependency Parsing (2020, methodology): “To control the amount of sharing across languages, we generate trainable parameters of the model by using a contextual parameter generator (CPG) function inspired by Platanios et al. (2018).”
A Study of Multilingual Neural Machine Translation (2019, background): “(Platanios et al. 2018) propose a novel contextual parameter generator to generate model for low/zero-resource language pairs.”
WeightNet: Revisiting the Design Space of Weight Networks (2020, methodology): “The methods [14,6,24,25] provide a worthy design space in the weight-generating network, our method follows the spirits and uses a WeightNet to generate the weights.”
Zero-Resource Neural Machine Translation with Monolingual Pivot Data (2019, methodology): “We do not employ any extensions to the zero-shot architecture (Arivazhagan et al., 2019; Lu et al., 2018; Platanios et al., 2018), although the meth-ods described here could easily be applied to such extensions as well.”
Multilingual Translation from Denoising Pre-Training (2021, background): “Subsequently, work focused on the the model capacity bottleneck, leading to exploration of various parameter sharing strategies (Blackwood et al., 2018; Platanios et al., 2018; Sachan and Neubig, 2018; Lu et al., 2018).”