Extreme Adaptation for Personalized Neural Machine Translation

By Paul Michel, Graham Neubig. 2018

113 citations recorded by the source corpus.

Every person speaks or writes their own flavor of their native language, influenced by a number of factors: the content they tend to talk about, their gender, their social status, or their geographical origin. When attempting to perform Machine Translation (MT), these variations have a significant effect on how the system should perform translation, but this is not captured well by standard one-size-fits-all models. In this paper, we propose a simple and parameter-efficient adaptation technique that only requires adapting the bias of the output softmax to each particular user of the MT system, either directly or through a factored approximation. Experiments on TED talks in three languages demonstrate improvements in translation accuracy, and better reflection of speaker traits in the target text.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Simple, Scalable Adaptation for Neural Machine Translation (2019, background): “Michel and Neubig (2018) proposed a space efﬁcient approach to adaptation that introduces domain-speciﬁc biases to the output vocabulary, enabling extreme personalization in settings where small amounts of data are available for a lot of different domains.”
BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models (2021, neutral): “Michel and Neubig (2018) ﬁne-tuned the biases of the output softmax in an NMT systems, to personalize the output vocabulary, and Frankle et al. (2020) have demonstrated that randomly-initialized CNNs achieve reasonable accuracy after training the batch-norm layers alone.”
Continual Learning for Neural Machine Translation (2021, methodology): “As for the bias in NMT systems, Michel and Neubig (2018) 2018 adapt the bias of the output softmax to build a personalized NMT model.”
Auditing Data Provenance in Text-Generation Models (2018, methodology): “The speaker annotated TED talks dataset (SATED) consists of transcripts from TED talks, 2 totaling 2,324 talks with roughly 271K sentences in each language [24].”
Contextual Parameter Generation for Universal Neural Machine Translation (2018, methodology): “For this reason, we decided to deﬁne the parameter generator networks as simple linear transforms, similar to the factored adaptation model of Michel and Neubig (2018), which was only applied to the bias terms of the output softmax: where l s , l t ∈ R M , W ( enc ) ∈ R P ( enc ) × M , W ( dec ) ∈…”
Reducing Gender Bias in Neural Machine Translation as a Domain Adaptation Problem (2020, neutral): “Recent work recognizes that NMT can be adapted to domains with desired attributes using small datasets (Farajian et al., 2017; Michel and Neubig, 2018).”
FRMT: A Benchmark for Few-Shot Region-Aware Machine Translation (2022, background): “, 2019); and personalized MT aims to match an individual’s specific style (Michel and Neubig, 2018; Vincent, 2021).”
Transfer Learning in Multilingual Neural Machine Translation with Dynamic Vocabulary (2018, methodology): “This ﬁne-tuning approach is prevalent in continued model trainings, for adapting NMT models [37, 38] or improving zero-shot and low-resource translation tasks [39, 40, 41].”
Freezing Subnetworks to Analyze Domain Adaptation in Neural Machine Translation (2018, background): “This suggests further work on adapting only a subset of parameters may be warranted (see also Vilar, 2018; Michel and Neubig, 2018).”
The Natural Auditor: How To Tell If Someone Used Your Words To Train Their Model (2018, methodology): “We use a sequence-to-sequence target model with the attention module as described in [26].”
Sequence to Sequence Mixture Model for Diverse Machine Translation (2018, background): “Michel and Neubig (2018) argue even without explicit signals (gender, politeness etc.), they can handle domain-specific information via annotation of speakers, and easily gain quality improvement from a larger number of domains.”
Case Study: Deontological Ethics in NLP (2020, background): “systems (Rabinovich et al., 2017; Michel and Neubig, 2018; Mirkin et al., 2015; Mirkin and Meunier, 2015) while preserving author attributes as well as controlling structural information like voice (Yamagishi et al.”
Pruning-then-Expanding Model for Domain Adaptation of Neural Machine Translation (2021, background): “The second category indcludes the following three systems: • Full Bias (Michel and Neubig, 2018) This method adds domain-specific bias term to the output softmax layer and only updates the term as other parts of the general-domain model keep fixed.”
Multiple-Attribute Text Rewriting (2018, methodology): “We also tried an approach similar to Michel & Neubig (2018), where the output layer of the decoder uses a different bias for each attribute label.”
The practical ethics of bias reduction in machine translation: why domain adaptation is better than data debiasing (2021, background): “For instance, adapting to small datasets with desired style, desired vocabulary, or simply reduced data noise has been previously explored (e.g., Farijian et al. 2017, Michel and Neubig 2018; Wang et al. 2018).”
Tool Learning with Foundation Models (2023, unknown): “Existing works cover a wide range of tasks, such as dialogue generation (Madotto et al., 2019; Mazaré et al., 2018; Song et al., 2021; Zhong et al., 2022), machine translation (Mirkin & Meunier, 2015; Michel & Neubig, 2018; Wuebker et al., 2018), and summarization (Yan et al., 2011).”