Gendered Ambiguous Pronoun (GAP) Shared Task at the Gender Bias in NLP Workshop 2019

By Kellie Webster, Marta Ruiz Costa-jussà, Christian Hardmeier, Will Radford. 2019

23 citations recorded by the source corpus.

The 1st ACL workshop on Gender Bias in Natural Language Processing included a shared task on gendered ambiguous pronoun (GAP) resolution. This task was based on the coreference challenge defined in Webster et al. (2018), designed to benchmark the ability of systems to resolve pronouns in real-world contexts in a gender-fair way. 263 teams competed via a Kaggle competition, with the winning system achieving logloss of 0.13667 and near gender parity. We review the approaches of eleven systems with accepted description papers, noting their effective use of BERT (Devlin et al., 2018), both via fine-tuning and for feature extraction, as well as ensembling.

Subfields: Artificial Intelligence, Natural Language Processing, AI Ethics, Fairness & Society

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yulia Tsvetkov
2. Kai-Wei Chang
3. Nanyun Peng
4. Emily Sheng
5. Anjalie Field
6. Sunipa Dev
7. Yejin Choi
8. Zeerak Talat

In Natural Language Processing

1. Kai-Wei Chang
2. Yulia Tsvetkov
3. Nanyun Peng
4. Emily Sheng
5. Sunipa Dev
6. Anjalie Field
7. Zeerak Talat
8. Jiao Sun

In AI Ethics, Fairness & Society

1. Yulia Tsvetkov
2. Zeerak Talat
3. Kai-Wei Chang
4. Jieyu Zhao
5. Su Lin Blodgett
6. Anjalie Field
7. Sunipa Dev
8. Bertie Vidgen

Citations in the corpus

• Language (Technology) is Power: A Critical Survey of “Bias” in NLP (2020, neutral): “…al. (2019); Précenth (2019); Pujari et al. (2019); Ruane et al. (2019); Stanovsky et al. (2019); Sun et al. (2019); Tan and Celis (2019); Webster et al. (2019); Zmigrod et al. (2019); Gyamfi et al. (2020); Hube et al. (2020); Hutchinson et al. (2020); Kim et al. (2020); Nadeem et al.…”
• A Survey of Race, Racism, and Anti-Racism in NLP (2021, background): “Work on NLP and race remains sparse, particularly in contrast to concerns about gender bias, which have led to surveys, workshops, and shared tasks (Sun et al., 2019; Webster et al., 2019).”
• Theories of “Gender” in NLP Bias Research (2022, background): “com/google-research[156] Kellie Webster, Marta R Costa-Jussà, Christian Hardmeier, and Will Radford.”
• On Measures of Biases and Harms in NLP (2021, background): “…in NLP typically have been categorized broadly into intrinsic or extrinsic evaluations based on whether they measure biased associations within the word embedding spaces (Caliskan et al., 2017) or biased decisions from models for specific tasks (Mohammad, 2018; Webster et al., 2019), respectively.”
• What do you mean, BERT? Assessing BERT as a Distributional Semantics Model (2019, methodology): “Using BERT representations has become in many cases a new standard approach: for instance, all submissions at the recent shared task on gendered pronoun resolution (Webster et al., 2019) were based on BERT .”
• Detecting Independent Pronoun Bias with Partially-Synthetic Data Generation (2020, background): “The recent Workshop on Gender Bias in Natural Language Processing (Costa-juss`a et al., 2019) had a shared task for English gender-ambiguous pro-nouns (Webster et al., 2018, 2019), but the dataset and task did not include possessive pronouns.”