Resolving Gendered Ambiguous Pronouns with BERT

By Kellie Webster, Marta Recasens, Vera Axelrod, Jason Baldridge. 2019

50 citations recorded by the source corpus.

Pronoun resolution is part of coreference resolution, the task of pairing an expression to its referring entity. This is an important task for natural language understanding and a necessary component of machine translation systems, chat bots and assistants. Neural machine learning systems perform far from ideally in this task, reaching as low as 73% F1 scores on modern benchmark datasets. Moreover, they tend to perform better for masculine pronouns than for feminine ones. Thus, the problem is both challenging and important for NLP researchers and practitioners. In this project, we describe our BERT-based approach to solving the problem of gender-balanced pronoun resolution. We are able to reach 92% F1 score and a much lower gender bias on the benchmark dataset shared by Google AI Language team.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Language (Technology) is Power: A Critical Survey of “Bias” in NLP (2020, methodology): “…here; most propose techniques we would have categorized as “Questionable correlations,” with a few as “Other representational harms” (Abzaliev, 2019; Attree, 2019; Bao and Qiao, 2019; Chada, 2019; Ionita et al., 2019; Liu, 2019; Lois et al., 2019; Wang, 2019; Xu and Yang, 2019; Yang et al., 2019).”
BBQ: A hand-built bias benchmark for question answering (2021, background): “The work often assesses model performance as a primary metric, with Webster et al. (2018) finding that existing biases in the training corpus led to models incorrectly adopting a bias towards selecting masculine pronouns.”
Theories of “Gender” in NLP Bias Research (2022, background): “pdf [65] Matei Ionita, Yury Kashnitsky, Ken Krige Kitsong, Vladimir Larin, Pjsc Sberbank, Denis Logvinenko, and Atanas Atanasov.”
Gender Bias in Multilingual Neural Machine Translation: The Architecture Matters (2020, background): “Among other strategies, several studies have proposed to work in data augmentation to balance data (Zmigrod et al., 2019) or forcing gender-balanced datasets (Webster et al., 2018; Costa-jussà et al., 2020).”
OSCaR: Orthogonal Subspace Correction and Rectification of Biases in Word Embeddings (2020, background): “But they are also known to capture a significant amount of societal biases [1, 6, 5, 17, 21] such as gender, race, nationality, or religion related biases, which gets expressed not just intrinsically but also downstream in the tasks that they are used for [5, 18, 20].”
“You Are Grounded!”: Latent Name Artifacts in Pre-trained Language Models (2020, background): “Bias can perpetuate to downstream tasks such as coreference resolution (Webster et al., 2018; Rudinger et al., 2018; Zhao et al., 2018), natural language inference (Rudinger et al., 2017), machine translation (Stanovsky et al., 2019), and sentiment analysis (Dı́az et al., 2018).”
BERT for Coreference Resolution: Baselines and Analysis (2019, methodology): “The base coreference model used in this paper from Lee et al. (2018) belongs to this family of models (Ng and Cardie, 2002; Bengtson and Roth, 2008; Denis and Baldridge, 2008; Fernandes et al., 2012; Durrett and Klein, 2013; Wiseman et al., 2015; Clark and Manning, 2016; Lee et al., 2017).”
Pre-training of Mention Representations in Coreference Models (2020, methodology): “, 2018; Kantor and Globerson, 2019; Joshi et al., 2019b). Lee et al. (2018) have refined the mention representation using attention over the antecedents of each mention. Kantor and Globerson (2019) showed the Entity Equalization approach to represent each mention in a cluster via an approximation of”
They, Them, Theirs: Rewriting with Gender-Neutral English (2021, background): “…Gonen and Goldberg, 2019) or whether there is the potential for harm when applied to downstream applications (Blodgett et al., 2020), including coreference resolution (Zhao et al., 2018; Rudinger et al., 2018; Webster et al., 2018) and language generation (Sheng et al., 2019; Nadeem et al., 2020).”
Collecting a Large-Scale Gender Bias Dataset for Coreference Resolution and Machine Translation (2021, background): “BUG is more than 24 times larger than the GAP coreference challenge set (Webster et al., 2018) and more than 30 times larger than WinoMT (Wino- Gender and Winobias combined) (Stanovsky et al., 2019).”
Adapting Coreference Resolution for Processing Violent Death Narratives (2021, background): “Researchers have shown coreference systems exhibit gender bias and resolve pronouns by relying heavily on gender information (Cao and Daumé III, 2020; Zhao et al., 2018; Rudinger et al., 2018; Webster et al., 2018; Zhao et al., 2019).”
Good Secretaries, Bad Truck Drivers? Occupational Gender Stereotypes in Sentiment Analysis (2019, background): “…such deep contextual representations (May et al., 2019; Basta et al., 2019) as well as their applications to coreference resolution (Zhao et al., 2019; Webster et al., 2018); however, no prior work has dealt with such models in the context of occupational gender stereotypes in sentiment analysis.”