Investigating Failures of Automatic Translation in the Case of Unambiguous Gender

By Adithya Renduchintala, Adina Williams. 2021

28 citations recorded by the source corpus.

Transformer-based models are the modern work horses for neural machine translation (NMT), reaching state of the art across several benchmarks. Despite their impressive accuracy, we observe a systemic and rudimentary class of errors made by current state-of-the-art NMT models with regards to translating from a language that doesn’t mark gender on nouns into others that do. We find that even when the surrounding context provides unambiguous evidence of the appropriate grammatical gender marking, no tested model was able to accurately gender occupation nouns systematically. We release an evaluation scheme and dataset for measuring the ability of NMT models to translate gender morphology correctly in unambiguous contexts across syntactically diverse sentences. Our dataset translates from an English source into 20 languages from several different language families. With the availability of this dataset, our hope is that the NMT community can iterate on solutions for this class of especially egregious errors.

Subfields: Artificial Intelligence, Gender Studies, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Gender Studies

In Natural Language Processing

Citations in the corpus

Societal Biases in Language Generation: Progress and Challenges (2021, background): “…(Escud´e Font and Costa-juss`a, 2019; Stanovsky et al., 2019; Saunders and Byrne, 2020; Saunders et al., 2020; Costa-juss`a and de Jorge, 2020; Basta et al., 2020; Stafanoviˇcs et al., 2020; Renduchintala and Williams, 2021; Choubey et al., 2021; Saunders et al., 2021; Tomalin et al., 2021).”
Hi, my name is Martha: Using names to measure and mitigate bias in generative dialogue models (2021, methodology): “To enable measurement, such works generally adopt linguistic proxies for gender, such as pronouns and/or occupations (Bolukbasi et al., 2016; Caliskan et al., 2017; Rudinger et al., 2018; Zhao et al., 2018; González et al., 2020; Renduchintala and Williams, 2021; de Vassimon Manela et al., 2021).”
Perturbation Augmentation for Fairer NLP (2022, background): “, 2019) or designing handcrafted grammars to generate perturbations (Zmigrod et al., 2019; Ma et al., 2021; Renduchintala and Williams, 2022; Papakipos and Bitton, 2022).”
Collecting a Large-Scale Gender Bias Dataset for Coreference Resolution and Machine Translation (2021, background): “Several works created synthetic datasets to evaluate gender bias (Kiritchenko and Mohammad, 2018; González et al., 2020; Renduchintala and Williams, 2021), e.”