RtGender: A Corpus for Studying Differential Responses to Gender

By Rob Voigt, David Jurgens, Vinodkumar Prabhakaran, Dan Jurafsky, Yulia Tsvetkov. 2018

81 citations recorded by the source corpus.

Like many social variables, gender pervasively inﬂuences how people communicate with one another. However, prior computational work has largely focused on linguistic gender difference and communications about gender, rather than communications directed to people of that gender, in part due to lack of data. Here, we ﬁll a critical need by introducing a multi-genre corpus of more than 25M comments from ﬁve socially and topically diverse sources tagged for the gender of the addressee. Using these data, we describe pilot studies on how differential responses to gender can be measured and analyzed and present 30k annotations for the sentiment and relevance of these responses, showing that across our datasets responses to women are more likely to be emotive and about the speaker as an individual (rather than about the content being responded to). Our dataset enables studying socially important questions like gender bias, and has potential uses for downstream applications such as dialogue systems, gender detection or obfuscation, and debiasing language generation.

Subfields: Artificial Intelligence, Gender Studies, Natural Language Processing, Evaluation & Benchmarks, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Gender Studies

In Natural Language Processing

In Evaluation & Benchmarks

In Safety, Robustness & Interpretability

Citations in the corpus

Social Biases in NLP Models as Barriers for Persons with Disabilities (2020, methodology): “We perform this method on a set of 1000 sentences extracted at random from the Reddit sub- 2Future work will see how to include non-binary pronouns. corpus of (Voigt et al., 2018).”
Adversarial Removal of Demographic Attributes from Text Data (2018, background): “…race, gender, and many other social factors of the author (Kop-pel et al., 2002; Burger et al., 2011; Nguyen et al., 2013; Weren et al., 2014; Verhoeven and Daele-mans, 2014; Rangel et al., 2016; Verhoeven et al., 2016; Blodgett et al., 2016), or even the audience of the text (Voigt et al., 2018).”
Perturbation Sensitivity Analysis to Detect Unintended Model Biases (2019, methodology): “Corpora: We use four socially and topically diverse corpora of online comments released by Voigt et al. (2018): Facebook comments on politi-cians’ posts (FB-Pol.) and on public ﬁgures’ posts (FB-Pub.)”
Politeness Transfer: A Tag and Generate Approach (2020, methodology): “For the Political slant task (Prabhumoye et al., 2018), we use dataset released by Voigt et al. (2018).”
Style Transfer Through Back-Translation (2018, background): “Our second dataset is comprised of top-level comments on Facebook posts from all 412 current members of the United States Senate and House who have public Facebook pages (Voigt et al., 2018).”
Detecting Emergent Intersectional Biases: Contextualized Word Embeddings Contain a Distribution of Human-like Biases (2020, background): “have shown that social biases are projected into Reddit comments [35].”
Unsupervised Discovery of Implicit Gender Bias (2020, methodology): “Our primary data is the Facebook subsection of the RtGender corpus (Voigt et al., 2018).”
Adhering, Steering, and Queering: Treatment of Gender in Natural Language Generation (2020, background): “Similar differences are identiﬁed in how people talk about men and women (e.g., [12,32,51,69]).”
Equity Beyond Bias in Language Technologies for Education (2019, background): “…et al., 2017; Garg et al., 2018) and is passed downstream, producing biased outcomes for tasks like coreference resolution (Zhao et al., 2018a), sentiment analysis (Kiritchenko and Mohammad, 2018), search (Romanov et al., 2019), and dialogue systems (Voigt et al., 2018; Henderson et al., 2018).”
Text Detoxification using Large Pre-trained Neural Models (2021, background): “The style transfer task is generally understood as rewriting of text with the same content and with altering of one or several attributes which constitute the “style”, such as authorship (Voigt et al., 2018), sentiment (Shen et al., 2017), or degree of politeness (Madaan et al., 2020).”
A Survey on Gender Bias in Natural Language Processing (2021, methodology): “Webster et al. [2018] find that existing resolvers do not perform well and are biased to favour better resolution of masculine pronouns.”
“Transforming” Delete, Retrieve, Generate Approach for Controlled Text Style Transfer (2019, methodology): “POLITICAL and GENDER: On these 2 datasets, (Prabhumoye et al., 2018) report state-of-art results using their model BT , which we similarly corroborate.”
Quantifying Gender Bias in Different Corpora (2020, methodology): “• RtGender RtGender [11] is a labeled multi-genre dataset that studies responses to online speech when the gender of the speaker is known.”
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜 (2021, background): “Furthermore, in addition to abusive language [139] and hate speech [67], there are subtler forms of negativity such as gender bias [137], microaggressions [22], dehumanization [83], and various socio-political framing biases [44, 114] that are prevalent in language data.”
Automatically Inferring Gender Associations from Language (2019, background): “There is also research studying how people talk to women and men (Voigt et al., 2018), as well as how people talk about women and men, typically in specific domains such as sports journalism (Fu et al.”
Deep Learning for Text Style Transfer: A Survey (2020, background): “The political slant dataset (Voigt et al. 2018) is collected from comments on Facebook posts of the United States Senate and House members.”
It’s going to be okay: Measuring Access to Support in Online Communities (2018, unknown): “Further, we hope that when used in combination with our support classifier and a larger context of gendered interactions (Voigt et al., 2018), these technologies can identify healthy communities that are supportive of all people.”
Civil Rephrases Of Toxic Texts With Self-Supervised Transformers (2021, background): “…al., 2011; He and McAuley, 2016; Shen et al., 2017; Li et al., 2018), but also addresses sociolinguistic datasets containing text in various registers (Gan et al., 2017; Rao and Tetreault, 2018) or with different identity markers (Voigt et al., 2018; Prabhu-moye et al., 2018; Lample et al., 2019).”