Hi, my name is Martha: Using names to measure and mitigate bias in generative dialogue models

By Eric Michael Smith, Adina Williams. 2021

31 citations recorded by the source corpus.

All AI models are susceptible to learning biases in data that they are trained on. For generative dialogue models, being trained on real human conversations containing unbalanced gender and race/ethnicity references can lead to models that display learned biases, which we define here broadly as any measurable differences in the distributions of words or semantic content of conversations based on demographic groups. We measure the strength of such biases by producing artificial conversations between two copies of a dialogue model, conditioning one conversational partner to state a name commonly associated with a certain gender and/or race/ethnicity. We find that larger capacity models tend to exhibit more gender bias and greater stereotyping of occupations by gender. We show that several methods of tuning these dialogue models, specifically name scrambling, controlled generation, and unlikelihood training, are effective in reducing bias in conversation, including on a downstream conversational task. Name scrambling is also effective in lowering differences in token usage across conversations where partners have names associated with different genders or races/ethnicities.

Subfields: Artificial Intelligence, Sociology and Political Science, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Sociology and Political Science

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, background): “, 2021b), but empirical work to validate this claim is still ongoing (Dodge et al., 2021; Smith and Williams, 2021; Tal et al., 2022), and further evidence from up-to-date models is still needed.”
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage (2022, background): “, 2021), including work from our group (Xu et al., 2020; Dinan et al., 2022, 2021; Smith et al., 2022; Dinan et al., 2020a; Smith and Williams, 2021).”
This Prompt is Measuring : Evaluating Bias Evaluation in Language Models (2023, background): “Although the appropriateness of demographic group and proxy choices unavoidably depend on cultural context, assumptions about such context are rarely explicitly stated; exceptions include Li et al. (2020) and Smith and Williams (2021).”
Perturbation Augmentation for Fairer NLP (2022, methodology): “One option to address any possible biases in names that result from crowdsourcing could be to run another post hoc stage of heuristic name perturbation (Smith and Williams, 2021) to ablate the contribution of demographic information on names alto-gether.”
On the Machine Learning of Ethical Judgments from Natural Language (2022, methodology): “…through analysis of the models, their training regimes, and the data that they rely on (Hall Maud-slay et al., 2019; Zhao et al., 2019; Dinan et al., 2020a,b; Vargas and Cotterell, 2020; Smith and Williams, 2021; Talat et al., 2021), rather than seeking to imbue models with a sense of ethics.”
"I'm sorry to hear that": finding bias in language models with a holistic descriptor dataset (2022, methodology): “…model has somewhat less bias than the larger 3B-parameter one, reﬂecting similar correlations between model size and bias in Bender et al. (2021) and Smith and Williams (2021), and the presence or absence of internet search in the 3B-parameter BlenderBot 2.0 model leaves the bias relatively…”
AugLy: Data Augmentations for Robustness (2022, background): “In NLP, there are studies that augment text to assess a model’s biases towards gender [31, 3] and ethnicity[26].”