Demographic Factors Improve Classification Performance

By Dirk Hovy. 2015

192 citations recorded by the source corpus.

Extra-linguistic factors influence language use, and are accounted for by speakers and listeners. Most natural language processing (NLP) tasks to date, however, treat language as uniform. This assumption can harm performance. We investigate the effect of including demographic information on performance in a variety of text-classification tasks. We find that by including age or gender information, we consistently and significantly improve performance over demographic-agnostic models. These results hold across three text-classification tasks in five languages.

Subfields: Artificial Intelligence, AI Ethics, Fairness & Society

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In AI Ethics, Fairness & Society

Citations in the corpus

Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systems (2018, background): “This paper does not examine the differences in accuracies of systems on text produced by different races or genders, as was done by Hovy (2015);”
Identifying and Reducing Gender Bias in Word-Level Language Models (2019, background): “Any dataset that has such demographic bias can lead to (potentially unintended) social exclusion (Hovy, 2015).”
The Social Impact of Natural Language Processing (2016, background): “…2011; Nguyen et al., 2011; Alowibdi et al., 2013; Ciot et al., 2013; Liu and Ruths, 2013; Volkova et al., 2014; Volkova et al., 2015; Plank and Hovy, 2015; Preotiuc-Pietro et al., 2015a; Preot¸iuc-Pietro et al., 2015b), and the characteristics in turn can be detected by and inﬂuence the…”
Predictive Biases in Natural Language Processing Models: A Conceptual Framework and Overview (2019, background): “Often the accuracy of text classifiers can be improved by considering the author demographics Hovy (2015); Lynn et al. (2017), which in turn could lead to decreased error disparity.”
Tagging Performance Correlates with Author Age (2015, background): “In the light of recent research (Volkova et al., 2013; Hovy, 2015; Jørgensen et al., 2015), we explore the hypothesis that these biases transfer to NLP tools induced from these resources.”
Women’s Syntactic Resilience and Men’s Grammatical Luck: Gender-Bias in Part-of-Speech Tagging and Dependency Parsing (2019, background): “While data bias can be overcome by accounting for demographics, and can even improve classification performance (Volkova et al., 2013; Hovy, 2015; Bolukbasi et al., 2016; Benton et al., 2017; Zhao et al., 2017; Lynn et al., 2017), there is still little”
Towards Robust and Privacy-preserving Text Representations (2018, methodology): “Data We again use the TrustPilot dataset derived from Hovy et al. (2015), however now we consider the RATING score as the target variable, not POS-tag.”
Personalized Machine Translation: Preserving Original Author Traits (2016, background): “While modeling of demographic traits has been proven beneﬁcial in some NLP tasks such as sentiment analysis (Volkova et al., 2013) or topic clas-siﬁcation (Hovy, 2015), very little attention has been paid to translation.”
Five sources of bias in natural language processing (2021, background): “Hovy (2015) and Jør-gensen et al. (2015) have shown the consequences of exclusion for various groups, for example, people under 35 and speakers of African-American vernacular English.”
The Importance of Modeling Social Factors of Language: Theory and Practice (2021, background): “Hovy (2016) showed that demographically-conditioned generated text also is more convincing.”
Multilingual Twitter Corpus and Baselines for Evaluating Demographic Bias in Hate Speech Recognition (2020, background): “Demographic factors can improve the performances of document classifiers (Hovy, 2015), and demographic variations root in language, especially in social media data (Volkova et al., 2013; Hovy, 2015).”
Towards Socially Responsible AI: Cognitive Bias-Aware Multi-Objective Learning (2020, background): “Hovy (2015) showed that the use of demographic features, such as age, and gender can improve the text classification task performance across five languages.”
The Role of Protected Class Word Lists in Bias Identification of Contextualized Word Representations (2019, background): “Demographic biases manifest in many ways, from disparities in tagging and classiﬁcation accuracy depending on author age and gender (Hovy, 2015; Dixon et al., 2018), to over-ampliﬁcation of demographic differences in language generation (Yatskar et al., 2016; Zhao et al., 2017), to diverging…”
Computational Sociolinguistics: A Survey (2016, background): “Hovy [121] found that training gender or age-specific word embeddings improved tasks such as sentiment analysis and topic classification.”
Semantic Variation in Online Communities of Practice (2018, unknown): “…the intricate ways in which patterns of word use and communities of individuals are related is essential for characterising the interests and the expressive means of sub-cultures, as well as to develop NLP tools that are effective in the face of variation (Hovy, 2015; Yang and Eisenstein, 2017).”
Motivating Personality-aware Machine Translation (2015, background): “Hovy (2015) extends this work to other languages and NLP tasks.”
CONAN - COunter NArratives through Nichesourcing: a Multilingual Dataset of Responses to Fight Online Hate Speech (2019, background): “…linguistic styles (Johannsen et al., 2015; Hovy and Spruit, 2016; van der Goot et al., 2018; DellOrletta and Nissim, 2018), improving text classiﬁcation (Mandel et al., 2012; Volkova et al., 2013; Hovy, 2015), or personalizing conversational agents (Qiu and Benbasat, 2010; Mazar´e et al., 2018a).”