Dats Wassup!!: Investigating African-American Vernacular English in Transformer-Based Text Generation

By Sophie Groenwold, Li-hsueh Ou, Aesha Parekh, Samhita Honnavalli, Sharon Levy, Diba Mirza, William Yang Wang. 2020

92 citations recorded by the source corpus.

The growth of social media has encouraged the written use of African American Vernacular English (AAVE), which has traditionally been used only in oral contexts. However, NLP models have historically been developed using dominant English varieties, such as Standard American English (SAE), due to text corpora availability. We investigate the performance of GPT-2 on AAVE text by creating a dataset of intent-equivalent parallel AAVE/SAE tweet pairs, thereby isolating syntactic structure and AAVE- or SAE-specific language for each pair. We evaluate each sample and its GPT-2 generated text with pretrained sentiment classifiers and find that while AAVE text results in more classifications of negative sentiment than SAE, the use of GPT-2 generally increases occurrences of positive sentiment for both. Additionally, we conduct human evaluation of AAVE and SAE text generated with GPT-2 to compare contextual rigor and overall quality.

Subfields: Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Detoxifying Language Models Risks Marginalizing Minority Voices (2021, methodology): “We use WAE tweets from Groenwold et al. (2020).”
Societal Biases in Language Generation: Progress and Challenges (2021, methodology): “There are also works that deﬁne biases towards people who produce the text (Groenwold et al., 2020) or people to whom the text is addressed (Sheng et al., 2021b), though there are relatively fewer works that study these latter associations.”
Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets (2021, background): “Groenwold et al., 2020), or pairs of free-form contrastive sentences (Nadeem et al., 2020; Nangia et al., 2020).”
An Empirical Study of Metrics to Measure Representational Harms in Pre-Trained Language Models (2023, background): “Existing works, measure representational harms in the response generated by the NLG system via automatic classifiers such as regard (Sheng et al., 2019), sentiment (Groenwold et al., 2020), style (Smith et al., 2020), and toxicity (Dhamala et al., 2021).”
Challenges in Automated Debiasing for Toxic Language Detection (2021, methodology): “For future investigations, we recommend obtaining human-written AAE - WAE pairs (e.g., as done by Groenwold et al., 2020).”
Characteristics of Harmful Text: Towards Rigorous Benchmarking of Language Models (2022, background): “TwitterAAE [15] and SAE/AAVE Pairs [44] explicitly measure the ability of models to generate language which aligns with a certain dialect, which could be seen as taking on a “persona” of some-one who speaks a dialect.”
A Survey of Race, Racism, and Anti-Racism in NLP (2021, background): “…et al. (2016) does serve as a starting point and forms the basis of most current work assessing performance gaps in NLP models (Sap et al., 2019; Blodgett et al., 2018; Xia et al., 2020; Xu et al., 2019; Groenwold et al., 2020), but even this corpus is explicitly not intended to infer race.”
What do Bias Measures Measure? (2021, background): “For example, Groenwold et al. (2020) use extracted African American English prompts to create a parallel set of White-Aligned English Twitter prompts and compare the sentiment of generated texts.”
On Measures of Biases and Harms in NLP (2021, background): “There are also other bias measures for language models that rely on sentiment (Groenwold et al., 2020; Shwartz et al., 2020).”
Exploring the Role of Grammar and Word Choice in Bias Toward African American English (AAE) in Hate Speech Classification (2022, background): “Further, one study found that AAE in general fares more negatively in sentiment analysis, and is more likely to be identified as angry or fearful sentiment than a Standard English equivalent sentence [16].”