Tagging Performance Correlates with Author Age

By Dirk Hovy, Anders Søgaard. 2015

135 citations recorded by the source corpus.

Many NLP tools for English and German are based on manually annotated articles from the Wall Street Journal and Frankfurter Rundschau. The average readers of these two newspapers are middle-aged (55 and 47 years old, respectively), and the annotated articles are more than 20 years old by now. This leads us to speculate whether tools induced from these resources (such as part-of-speech taggers) put older language users at an advantage. We show that this is actually the case in both languages, and that the cause goes beyond simple vocabulary differences. In our experiments, we control for gender and region.

Subfields: Artificial Intelligence, Natural Language Processing, Social Sciences & Humanities

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Social Sciences & Humanities

Citations in the corpus

Language (Technology) is Power: A Critical Survey of “Bias” in NLP (2020, background): “Other representational harms Hovy and Søgaard (2015); Blodgett et al. (2016); Bolukbasi et al. (2016b); Hovy and Spruit (2016); Blodgett and O’Connor (2017); Larson (2017); Schnoebelen (2017); Blodgett et al. (2018); Curry and Rieser (2018); Díaz et al. (2018); Dixon et al. (2018); Kiritchenko and…”
Reducing Sentiment Bias in Language Models via Counterfactual Evaluation (2019, background): “Besides learning to favor the language of the authors’ demographic group (Hovy and Søgaard, 2015), NLP models can pick up on a variety of cultural associations and undesirable social biases (Caliskan et al.”
Data Statements for Natural Language Processing: Toward Mitigating System Bias and Enabling Better Science (2018, background): “There are other studies showing that systems from part of speech taggers (Hovy and Søgaard, 2015; Jørgensen et al., 2015) to speech recognition engines (Tatman, 2017) perform better for speakers whose demographic characteristics better match those represented in the training data.”
The Social Impact of Natural Language Processing (2016, background): “Concretely, the consequences of exclusion for NLP research have recently been pointed out by Hovy and Søgaard (2015) and Jørgensen et al. (2015): current state-of-the-art NLP models score a signiﬁcantly lower accuracy for young people and ethnic minorities vis-`a-vis the modeled demo-graphics.”
Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation (2017, methodology): “…impetus to the exploration of non-canonical language (e.g. Eisenstein, 2013); the impact of social and demographic factors on language use (e.g. Hovy & Søgaard, 2015; Johannsen et al., 2015); the prevalence of paralinguistic features such as affect, irony and humour (Pang & Lee, 2008; Lukin &…”
Predictive Biases in Natural Language Processing Models: A Conceptual Framework and Overview (2019, background): “tion bias (Hovy and Søgaard, 2015; Jørgensen et al., 2015).”
Challenges of studying and processing dialects in social media (2015, background): “A number of recent papers have explored social media with respect to sociolinguistic and dialectological questions (Rao et al., 2010; Eisenstein, 2013; Volkova et al., 2013; Doyle, 2014; Hovy et al., 2015; Volkova et al., 2015; Johannsen et al., 2015; Hovy and Søgaard, 2015; Eisenstein, to appear).”
Women’s Syntactic Resilience and Men’s Grammatical Luck: Gender-Bias in Part-of-Speech Tagging and Dependency Parsing (2019, background): “, 2017), which in turn can be detrimental to members of the underrepresented demographic groups (Jørgensen et al., 2015; Hovy and Søgaard, 2015; Hovy and Spruit, 2016).”
Data Statements for NLP: Toward Mitigating System Bias and Enabling Better Science (2018, background): “There are other studies showing that systems from part of speech taggers (Hovy and Søgaard, 2015; Jørgensen et al., 2015) to speech recognition engines (Tatman, 2017) perform”
Analyzing Biases in Human Perception of User Age and Gender from Text (2016, background): “Hovy and Søgaard (2015) showed that the author age is a factor inﬂuencing training part-of-speech taggers.”
Computational Sociolinguistics: A Survey (2016, background): “Recently, various studies (Hovy 2015; Stoop and van den Bosch 2014; Volkova, Wilson, and Yarowsky 2013) have demonstrated that existing NLP tools can be improved by accounting for linguistic variation due to social factors, and Hovy and Søgaard (2015) have drawn attention to the fact that biases in ”
Darling or Babygirl ? Investigating Stylistic Bias in Sentiment Analysis (2018, background): “Hovy & Søgaard (2015) find that part-of-speech taggers trained on traditional news corpora exhibit better performance on text written by older authors.”
It’s going to be okay: Measuring Access to Support in Online Communities (2018, unknown): “tence construction and length (Hovy and Søgaard, 2015), and a lexicon for age of acquisition (Kuperman et al.”
Mitigating Gender Bias in Machine Translation with Target Gender Annotations (2020, background): “…NLP resources and systems also suffer from the same socially constructed biases, as well as inaccuracies and incompleteness (Jørgensen et al., 2015; Hovy and Søgaard, 2015; Prates et al., 2019; Vanmassenhove et al., 2019; Bordia and Bowman, 2019; Davidson et al., 2019; Tan and Celis, 2019).”
Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations (2022, background): “Furthermore, spurious correlations can disproportionately affect different population groups (each with their own data distribution), leading to poor performance on the minority groups and unfair decisions [13, 39, 10, 94, 32, 20, 17, 79].”
From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models (2023, background): “Impact of Model and Data Bias on Downstream Task Fairness Previous research has shown that the performance of models for downstream tasks can vary greatly among different identity groups (Hovy and Søgaard, 2015; Buolamwini and Gebru, 2018; Dixon et al., 2018), highlighting the issue of fairness (Hut”
Quantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness Metrics (2021, background): “…metric, e.g., F1 or accuracy, or a more elaborate measure independently for each protected group or for each split of a challenge dataset (Hovy and Søgaard, 2015; Rudinger et al., 2018; Zhao et al., 2018; Garimella et al., 2019; Sap et al., 2019; Bagdasaryan et al., 2019; Stafanoviˇcs et…”
Cross-lingual syntactic variation over age and gender (2015, background): “Volkova et al. (2013), Hovy and Søgaard (2015), Jørgensen et al. (2015), and Hovy (2015) have shown the impact of demographic factors on NLP performance.”
Square One Bias in NLP: Towards a Multi-Dimensional Exploration of the Research Manifold (2022, background): “Annotation biases that stem from annotator demographics have been studied for English POS tagging (Hovy and Søgaard, 2015) or English summarization (Jørgensen and Søgaard,”