Analyzing Biases in Human Perception of User Age and Gender from Text

By Lucie Flekova, Jordan Carpenter, Salvatore Giorgi, Lyle Ungar, Daniel Preotiuc-Pietro. 2016

54 citations recorded by the source corpus.

User traits disclosed through written text, such as age and gender, can be used to per-sonalize applications such as recommender systems or conversational agents. However, human perception of these traits is not perfectly aligned with reality. In this paper, we conduct a large-scale crowdsourcing ex-periment on guessing age and gender from tweets. We systematically analyze the quality and possible biases of these predictions. We identify the textual cues which lead to miss-assessments of traits or make annotators more or less conﬁdent in their choice. Our study demonstrates that differences be-tween real and perceived traits are noteworthy and elucidates inaccurately used stereotypes in human perception.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

CHBias: Bias Evaluation and Mitigation of Chinese Conversational Language Models (2023, methodology): “…and attribute terms to cover more biases in texts, we collect target and attribute terms according to many previous NLP works on social biases (Nangia et al., 2020; Flekova et al., 2016; Barikeri et al., 2021), as well as sociology literature (Greenwald et al., 1998; Rhode, 2010; Krekula, 2007).”
Beyond Binary Labels: Political Ideology Prediction of Twitter Users (2017, background): “Applications include prediction of age (Rao et al., 2010; Flekova et al., 2016b), gender (Burger et al., 2011; Sap et al., 2014), personality (Schwartz et al., 2013; Preoţiuc-Pietro et al., 2016), socioeconomic status (Preoţiuc-Pietro et al., 2015a,b; Liu et al., 2016c), popularity (Lampos et…”
Predicting Authorship and Author Traits from Keystroke Dynamics (2018, methodology): “, 2011) with linear kernel and `2 regularization, similar to the state-ofthe-art in author profiling (Flekova et al., 2016a; Basile et al., 2017).”
RtGender: A Corpus for Studying Differential Responses to Gender (2018, background): “However, previous computational work on language and gender has focused mainly on language about or portraying persons of a particular gender (Wagner et al., 2015; Flekova et al., 2016; Agarwal et al., 2015).”
Adversarial Removal of Demographic Attributes Revisited (2019, result): “This is contrary to the ﬁndings of Flekova et al. (2016), but is similar by the unaveraged results of Burger et al. (2011).”
Toward Gender-Inclusive Coreference Resolution: An Analysis of Gender and Bias Throughout the Machine Learning Lifecycle* (2021, unknown): “(2016) N N Y Y − Y Y N Flekova et al. (2016) N N Y Y − Y Y N Tran and Ostendorf (2016) N N N Y − − − − Qian, Qiu, and Huang (2016) N Y N Y − − − − Li et al. (2016) N N Y Y − Y Y N Zhang et al. (2016) N N Y Y − Y Y N Garimella and Mihalcea (2016) N N Y Y − Y Y N Reddy and Knight (2016) N N Y Y − Y Y ”
User-Level Race and Ethnicity Predictors from Twitter Text (2018, unknown): “…work on predicting user traits from text (Burger et al., 2011; Rao et al., 2010; Pennacchiotti and Popescu, 2011; Schwartz et al., 2013; Sap et al., 2014; Volkova et al., 2014; Preoţiuc-Pietro et al., 2015b; Flekova et al., 2016b; Preoţiuc-Pietro et al., 2016; Preoţiuc-Pietro et al., 2017).”
Bleaching Text: Abstract Features for Cross-lingual Gender Prediction (2018, background): “Flekova et al. (2016) found that people tend to rely too much on stereotypical lexical indicators, while Nguyen et al. (2014) show that more than 10% of the Twitter users do actually not employ words that the crowd associates with their biological sex.”
Few-shot Instruction Prompts for Pretrained Language Models to Detect Social Biases (2021, background): “As shown in recent work, human annotation process can be a source of bias in itself (Hovy and Prabhumoye, 2021) and labelers may even be selected by malicious actors to introduce biased interpretation on purpose (Flekova et al., 2016).”