Women’s Syntactic Resilience and Men’s Grammatical Luck: Gender-Bias in Part-of-Speech Tagging and Dependency Parsing

By Aparna Garimella, Carmen Banea, Eduard H. Hovy, Rada Mihalcea. 2019

64 citations recorded by the source corpus.

Several linguistic studies have shown the prevalence of various lexical and grammatical patterns in texts authored by a person of a particular gender, but models for part-of-speech tagging and dependency parsing have still not adapted to account for these differences. To address this, we annotate the Wall Street Journal part of the Penn Treebank with the gender information of the articles’ authors, and build taggers and parsers trained on this data that show performance differences in text written by men and women. Further analyses reveal numerous part-of-speech tags and syntactic relations whose prediction performances benefit from the prevalence of a specific gender in the training data. The results underscore the importance of accounting for gendered differences in syntactic tasks, and outline future venues for developing more accurate taggers and parsers. We release our data to the research community.

Subfields: Artificial Intelligence, Gender Studies, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Gender Studies

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Language (Technology) is Power: A Critical Survey of “Bias” in NLP (2020, background): “…al. (2018); Badjatiya et al. (2019); Bhargava and Forsyth (2019); Cao and Daumé (2019); Cho et al. (2019); Davidson et al. (2019); Dev et al. (2019); Garimella et al. (2019); Garg et al. (2019); Huang et al. (2019); James-Sorenson and AlvarezMelis (2019); Kaneko and Bollegala (2019); Liu et al.…”
Queens Are Powerful Too: Mitigating Gender Bias in Dialogue Generation (2019, background): “2019), uncovering many gender disparities and harmful biases in algorithms and text (Cao and Daumé III 2020; Chang et al. 2019; Chang and McKeown 2019; Costa-jussà 2019; Du et al. 2019; Emami et al. 2019; Garimella et al. 2019; Gaut et al. 2020; Habash et al. 2019; Hashempour 2019; Hoyle et al. 2019”
A Survey on Bias and Fairness in Natural Language Processing (2022, neutral): “A famous example being the “Wall Street Journal effect”, where syntactic parsers and part-of-speech taggers perform most accurately over language written by middle-aged white men (Garimella et al., 2019).”
Predictive Biases in Natural Language Processing Models: A Conceptual Framework and Overview (2019, methodology): “The work by Garimella et al. (2019) has quantified this bias further by studying the effect of different gender compositions of the train- ing data on tagging and parsing, supporting the claim that debiased samples benefit performance.”
Investigating Failures of Automatic Translation in the Case of Unambiguous Gender (2021, neutral): “…in NLP and adjacent fields (Bolukbasi et al., 2016; Hovy and Spruit, 2016; Caliskan et al., 2017; Rudinger et al., 2017; Garg et al., 2018; Garimella et al., 2019; Gonen and Goldberg, 2019; Dinan et al., 2020a,b), we further observe in our results yet a higher order type of stereotyping…”
Sex and gender differences and biases in artificial intelligence for biomedicine and healthcare (2020, background): “In fact, major differences are observed in dialogue structure 75 , word reading 76 , and even in children ’ s linguistic tasks 77 .”
What do Bias Measures Measure? (2021, background): “For part-of-speech tagging, Munro and Morrison (2020) and Garimella et al. (2019) ﬁnd that state-of-the-art parsers perform differently across genders, failing to identify “hers” and “theirs” as pronouns but not “his”.”
Five sources of bias in natural language processing (2021, background): “…news sources (e.g., Wall Street Journal, Frankfurter Rundschau from the 1980s through the 1990s), a very codified domain predominantly produced by a small, homogeneous sample: typically white, middle-aged, educated, upper-middle-class men (Garimella et al., 2019; Hovy & Søgaard, 2015).”
Gender Bias in Machine Translation (2021, methodology): “Drawing on gender-related features proved useful for building demographically informed NLP tools (Garimella et al., 2019) and personalized MT models (Mirkin et al., 2015; Bawden et al., 2016; Rabinovich et al., 2017).”
A Survey on Gender Bias in Natural Language Processing (2021, methodology): “2020], and part-of-speech tagging and parsing [Garimella et al. 2019].”
On Measures of Biases and Harms in NLP (2021, background): “For part-of-speech tagging, Munro and Morrison (2020) and Garimella et al. (2019) find that state-of-the-art parsers perform differently across genders, failing to identify “hers” and “theirs” as pronouns but not “his”.”