Type B Reflexivization as an Unambiguous Testbed for Multilingual Multi-Task Gender Bias

By Ana Valeria González, Maria Barrett, Rasmus Hvingelby, Kellie Webster, Anders Søgaard. 2020

27 citations recorded by the source corpus.

The one-sided focus on English in previous studies of gender bias in NLP misses out on opportunities in other languages: English challenge datasets such as GAP and WinoGender highlight model preferences that are "hallucinatory", e.g., disambiguating gender-ambiguous occurrences of 'doctor' as male doctors. We show that for languages with type B reflexivization, e.g., Swedish and Russian, we can construct multi-task challenge datasets for detecting gender bias that lead to unambiguously wrong model predictions: In these languages, the direct translation of 'the doctor removed his mask' is not ambiguous between a coreferential reading and a disjoint reading. Instead, the coreferential reading requires a non-gendered pronoun, and the gendered, possessive pronouns are anti-reflexive. We present a multilingual, multi-task challenge dataset, which spans four languages and four NLP tasks and focuses only on this phenomenon. We find evidence for gender bias across all task-language combinations and correlate model bias with national labor market statistics.

Subfields: Artificial Intelligence, Gender Studies, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Gender Studies

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Hi, my name is Martha: Using names to measure and mitigate bias in generative dialogue models (2021, methodology): “To enable measurement, such works generally adopt linguistic proxies for gender, such as pronouns and/or occupations (Bolukbasi et al., 2016; Caliskan et al., 2017; Rudinger et al., 2018; Zhao et al., 2018; González et al., 2020; Renduchintala and Williams, 2021; de Vassimon Manela et al., 2021).”
Investigating Failures of Automatic Translation in the Case of Unambiguous Gender (2021, methodology): “Recently, several works (Stanovsky et al., 2019; Prates et al., 2019; Gonen and Webster, 2020; González et al., 2020) investigated gender bias in multiple languages with complex morphology, and showed that state-of-the-art MT systems resolve gender-unbalanced occupation nouns (from the US Bureau of ”
Collecting a Large-Scale Gender Bias Dataset for Coreference Resolution and Machine Translation (2021, background): “Several works created synthetic datasets to evaluate gender bias (Kiritchenko and Mohammad, 2018; González et al., 2020; Renduchintala and Williams, 2021), e.”
Sometimes We Want Ungrammatical Translations (2021, background): “…input to an output that is not faithful to the source, and sometimes is even factually incorrect (Vinyals and Le, 2015; Koehn and Knowles, 2017; Wiseman et al., 2017; Nie et al., 2019; Kryscinski et al., 2020; Maynez et al., 2020; Tian et al., 2020; González et al., 2020; Xiao and Wang, 2021).”
Square One Bias in NLP: Towards a Multi-Dimensional Exploration of the Research Manifold (2022, background): “González et al. (2020) also study gender bias, but by relying on reflexive pronominal constructions that do not exist in the English language; this is a good example of research that would not have been possible taking SQUARE ONE as our point of departure.”