Reducing Gender Bias in Neural Machine Translation as a Domain Adaptation Problem

By Danielle Saunders, Bill Byrne. 2020

158 citations recorded by the source corpus.

Training data for NLP tasks often exhibits gender bias in that fewer sentences refer to women than to men. In Neural Machine Translation (NMT) gender bias has been shown to reduce translation quality, particularly when the target language has grammatical gender. The recent WinoMT challenge set allows us to measure this effect directly (Stanovsky et al, 2019) Ideally we would reduce system bias by simply debiasing all data prior to training, but achieving this effectively is itself a challenge. Rather than attempt to create a ‘balanced’ dataset, we use transfer learning on a small set of trusted, gender-balanced examples. This approach gives strong and consistent improvements in gender debiasing with much less computational cost than training from scratch. A known pitfall of transfer learning on new domains is ‘catastrophic forgetting’, which we address at adaptation and inference time. During adaptation we show that Elastic Weight Consolidation allows a performance trade-off between general translation quality and bias reduction. At inference time we propose a lattice-rescoring scheme which outperforms all systems evaluated in Stanovsky et al, 2019 on WinoMT with no degradation of general test set BLEU. We demonstrate our approach translating from English into three languages with varied linguistic properties and data availability.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Language (Technology) is Power: A Critical Survey of “Bias” in NLP (2020, background): “” —Saunders and Byrne (2020)”
Societal Biases in Language Generation: Progress and Challenges (2021, methodology): “…Font and Costa-juss`a, 2019; Elaraby et al., 2018; Habash et al., 2019; Alhafni et al., 2020) or accuracy on WinoMT (Saunders and Byrne, 2020; Saunders et al., 2020; Kocmi et al., 2020; Costa-juss`a and de Jorge, 2020; Costa-juss`a et al., 2020; Basta et al., 2020; Choubey et al., 2021;…”
Avoiding catastrophic forgetting in mitigating model biases in sentence-pair classification with elastic weight consolidation (2020, methodology): “debiasing as domain adaptation with EWC was successful in mitigating gender bias in NMT by Saunders and Byrne (2020).”
Gender Bias in Machine Translation (2021, methodology): “In fact, unlike (Saunders and Byrne, 2020), implementing the re-inflection component demanded the expensive work of data creation of the Arabic Parallel Gender Corpus (§4.”
Five sources of bias in natural language processing (2021, background): “Systems that explicitly model user demographics will help produce both more personalized and less biased translations (Font & Costa-jussà, 2019; Mirkin et al., 2015; Mirkin & Meuni-er, 2015; Saunders & Byrne, 2020; Stanovsky et al., 2019).”
Mitigating Gender Bias in Machine Translation with Target Gender Annotations (2020, methodology): “Saunders and Byrne (2020) describe methods for NMT model adaptation using a hand-crafted gender-balanced dataset and a translation re-scoring scheme based on the adapted models.”
Detect and Perturb: Neutral Rewriting of Biased and Sensitive Text via Gradient-based Decoding (2021, unknown): “Saunders and Byrne (2020) mitigate gender bias in machine translation via transfer learning using handcrafted gender-balanced datasets.”
First the Worst: Finding Better Gender Translations During Beam Search (2021, unknown): “, 2020) or tuning (Saunders and Byrne, 2020; Basta et al., 2020) on gender-adjusted data.”
Improving Gender Translation Accuracy with Filtered Self-Training (2021, unknown): “In machine translation, Saunders and Byrne (2020) showed that gender translation accuracy for unambiguous inputs can be improved through ﬁne-tuning on small gender-balanced but counterfactual data.”
Gender bias amplification during Speed-Quality optimization in Neural Machine Translation (2021, unknown): “Vanmassenhove et al. (2018), Stafanoviˇcs et al. (2020) and Saunders et al. (2020) propose a data-annotation scheme in which the NMT model is trained to obey gender-speciﬁc tags provided with the source sentence.”
The practical ethics of bias reduction in machine translation: why domain adaptation is better than data debiasing (2021, background): “In addition, recent work by Saunders et al (2020) extends some of the adaptation work mentioned in this article.”
Towards Cross-Lingual Generalization of Translation Gender Bias (2021, background): “In general, the languages with more resources 23 recorded better fluency index in to-English translation.”
Neural Machine Translation Doesn’t Translate Gender Coreference Right Unless You Make It (2020, methodology): “Vanmassenhove et al. (2018) incorporate a ‘speaker gender’ tag into training data, allowing gender to be conveyed at the sentence level.”
A Survey on Gender Bias in Natural Language Processing (2021, methodology): “To the best of our knowledge, Schmidt [2015] propose the first word embedding debiasing algorithm and remove multiple gender dimensions fromword vectors. In parallel, instead of completely removing gender information, Bolukbasi et al. [2016] suggest shifting word embeddings to be equally male and fe”
Quantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness Metrics (2021, unknown): “…10 (Beutel et al., 2017), 11 (Blodgett and O’Connor, 2017; Bhaskaran and Bhallamudi, 2019), 12 (De-Arteaga et al., 2019)„ 13 (Stanovsky et al., 2019; Saunders and Byrne, 2020), 14 (Blodgett et al., 2018), 15 (Bamman et al., 2019), 16 (Webster et al., 2018), 17 (Garg et al., 2019), 18 19 20…”
Fine-tuning Neural Machine Translation on Gender-Balanced Datasets (2020, background): “The closest work to ours is the one by Saunders and Byrne (2020) where authors generate a small gender-balanced dataset and use Elastic Weight Consolidation techniques to perform transfer learning and mitigate the consequences of training with unbalanced datasets.”
Decoding and Diversity in Machine Translation (2020, background): “It has been observed that NMT can often produce misgendered outputs, and solutions focusing on the modeling stage have been proposed, however, our study implicates search as a source of gender pronoun bias (Vanmassenhove et al., 2018; Saunders and Byrne, 2020).”