Challenges of studying and processing dialects in social media

By Anna Katrine van Zee, Dirk Hovy, Anders Søgaard. 2015

86 citations recorded by the source corpus.

Dialect features typically do not make it into formal writing, but flourish in social media. This enables largescale variational studies. We focus on three phonological features of African American Vernacular English and their manifestation as spelling variations on Twitter. We discuss to what extent our data can be used to falsify eight sociolinguistic hypotheses. To go beyond the spelling level, we require automatic analysis such as POS tagging, but social media language still challenges language technologies. We show how both newswire- and Twitter-adapted stateof-the-art POS taggers perform significantly worse on AAVE tweets, suggesting that large-scale dialect studies of language variation beyond the surface level are not feasible with out-ofthe-box NLP tools.

Subfields: Artificial Intelligence, Language and Linguistics, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Language and Linguistics

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Language (Technology) is Power: A Critical Survey of “Bias” in NLP (2020, background): “Other representational harms Jørgensen et al. (2015); Hovy and Søgaard (2015); Blodgett et al. (2016); Blodgett and O’Connor (2017); Blodgett et al. (2018); Curry and Rieser (2018); Dixon et al. (2018); Park et al. (2018); Thelwall (2018); Webster et al. (2018); Badjatiya et al. (2019); Bagdasaryan…”
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “…using large web-scraped datasets, and to Luccioni and Viviano (2021) and Dodge et al. (2021) for analyses of such large-scale web-scraped corpora and their representational, legal, consent, and PII issues. text with AAVE features (Jørgensen et al., 2015, 2016; Blodgett et al., 2016, among others).”
Data Statements for Natural Language Processing: Toward Mitigating System Bias and Enabling Better Science (2018, background): “There are other studies showing that systems from part of speech taggers (Hovy and Søgaard, 2015; Jørgensen et al., 2015) to speech recognition engines (Tatman, 2017) perform better for speakers whose demographic characteristics better match those represented in the training data.”
The Social Impact of Natural Language Processing (2016, background): “Concretely, the consequences of exclusion for NLP research have recently been pointed out by Hovy and Søgaard (2015) and Jørgensen et al. (2015): current state-of-the-art NLP models score a signiﬁcantly lower accuracy for young people and ethnic minorities vis-`a-vis the modeled demo-graphics.”
Demographic Dialectal Variation in Social Media: A Case Study of African-American English (2016, methodology): “To search for the constructions, we tagged the corpora using the ARK Twitter POS tagger (Gimpel et al., 2011; Owoputi et al., 2013), 9 which Jørgensen et al. (2015) show has similar accuracy rates on both AAE and non-AAE tweets, unlike other POS tag-gers.”
Predictive Biases in Natural Language Processing Models: A Conceptual Framework and Overview (2019, background): “Within NLP, some of the first works to note demographic biases were due to a selection bias (Hovy and Søgaard, 2015; Jørgensen et al., 2015).”
Style Transfer Through Back-Translation (2018, background): “Moreover, prior work has shown that the quality of language identification and POS tagging degrades significantly on African American Vernacular English (Blodgett et al., 2016; Jørgensen et al., 2015); YouTube’s automatic captions have higher error rates for women and speakers from Scotland (Rudinge”
WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020, neutral): “…have been shown to obtain worse performance on African-American Vernacular English compared to Standard American English on part-of-speech tagging (Jørgensen et al., 2015), dependency parsing (Blodgett et al., 2016), language identification (Blodgett and O’Connor, 2017), and auto-correct systems…”
Racial Disparity in Natural Language Processing: A Case Study of Social Media African-American English (2017, background): “In the rest of this paper, we focus on the dialect of African-American English as used on Twitter, which previous work [3, 9, 11] has established is very prevalent and sometimes quite different than mainstream American English.”
Tagging Performance Correlates with Author Age (2015, background): “In the light of recent research (Volkova et al., 2013; Hovy, 2015; Jørgensen et al., 2015), we explore the hypothesis that these biases transfer to NLP tools induced from these resources.”
Women’s Syntactic Resilience and Men’s Grammatical Luck: Gender-Bias in Part-of-Speech Tagging and Dependency Parsing (2019, background): “Such demographic imbalances are amplified by the model (Zhao et al., 2017), which in turn can be detrimental to members of the underrepresented demographic groups (Jørgensen et al., 2015; Hovy and Søgaard, 2015; Hovy and Spruit, 2016).”
Learning a POS tagger for AAVE-like language (2016, methodology): “In contrast, very little data is available for domains such as subtitles, lyrics and tweets — especially for language varieties such as AAVE.”
Equity Beyond Bias in Language Technologies for Education (2019, background): “Systems today may also fail to recognize speech from students speaking certain dialects or accents, though progress in recognition for marginalized language variation is improving rapidly (Blodgett et al., 2016; Stewart, 2014; Jørgensen et al., 2015).”
Dialect Diversity in Text Summarization on Twitter (2020, background): “However, dialect labels are not always available (or even desirable [5]) and automated dialect classification is a difficult task [35].”
Five sources of bias in natural language processing (2021, methodology): “As a consequence, NLP tools trained on one demographic sample perform worse on another sample (Ga-rimella et al., 2019; Hovy & Søgaard, 2015; Jørgensen et al., 2015).”
Analyzing Stereotypes in Generative Text Inference Tasks (2021, background): “…that resides in the heads of individual people—as well as a cultural phenomenon—that “[sterotypes] exist also in ‘the fabric of society’ itself” (Stangor and Schaller, 2012), and as such who the annotators are matters (Hovy and Spruit, 2016; Jørgensen et al., 2015; Hazen et al., 2020).”
Rissanen Data Analysis: Examining Dataset Characteristics via Description Length (2021, background): “…prior work found that machine learning systems are worse at classifying images of women (Phillips et al., 2000; Buolamwini and Gebru, 2018), at speech recognition for women and speakers from Scotland (Tatman, 2017), and at POS tagging for African American vernacular (Jørgensen et al., 2015).”
Dialect-robust Evaluation of Generated Text (2022, methodology): “A related line of work has focused on inducing dialect features from corpora (Eisenstein et al., 2010; Jørgensen et al., 2015; Dunn, 2021) and on recognizing dialect features in text (Demszky et al., 2021; Masis et al., 2022).”
User-Level Race and Ethnicity Predictors from Twitter Text (2018, unknown): “Part-of-Speech Tags We analyze part-of-speech tag usage across groups by POS tagging all tweets using the Twitter model of the Stanford Tagger (Derczynski et al., 2013), which showed best tagging results for AAVE (Jørgensen et al., 2015) and also uses the finer grained Penn Treebank tagset.”
Mitigating Gender Bias in Machine Translation with Target Gender Annotations (2020, background): “Thus the resulting NLP resources and systems also suffer from the same socially constructed biases, as well as inaccuracies and incompleteness (Jørgensen et al., 2015; Hovy and Søgaard, 2015; Prates et al., 2019; Vanmassenhove et al., 2019; Bordia and Bowman, 2019; Davidson et al., 2019; Tan and…”