On Transferability of Bias Mitigation Effects in Language Model Fine-Tuning

By Xisen Jin, Francesco Barbieri, Brendan Kennedy, Aida Mostafazadeh Davani, Leonardo Neves, Xiang Ren. 2021

84 citations recorded by the source corpus.

Fine-tuned language models have been shown to exhibit biases against protected groups in a host of modeling tasks such as text classification and coreference resolution. Previous works focus on detecting these biases, reducing bias in data representations, and using auxiliary training objectives to mitigate bias during fine-tuning. Although these techniques achieve bias reduction for the task and domain at hand, the effects of bias mitigation may not directly transfer to new tasks, requiring additional data collection and customized annotation of sensitive attributes, and re-evaluation of appropriate fairness metrics. We explore the feasibility and benefits of upstream bias mitigation (UBM) for reducing bias on downstream tasks, by first applying bias mitigation to an upstream model through fine-tuning and subsequently using it for downstream fine-tuning. We find, in extensive experiments across hate speech detection, toxicity detection and coreference resolution tasks over various bias factors, that the effects of UBM are indeed transferable to new downstream tasks or domains via fine-tuning, creating less biased downstream models than directly fine-tuning on the downstream task or transferring from a vanilla upstream model. Though challenges remain, we show that UBM promises more efficient and accessible bias mitigation in LM fine-tuning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Alignment & RLHF

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Alignment & RLHF

Citations in the corpus

“I’m sorry to hear that”: Finding New Biases in Language Models with a Holistic Descriptor Dataset (2022, background): “Despite the fact that upstream bias mitigations can transfer to extrinsic, “downstream”, tasks well (Jin et al., 2021), it is currently unclear whether intrinsic measurement is sufficient, in particular because intrinsic and extrinsic task-based bias metrics don’t always correlate (Delobelle et…”
Characteristics of Harmful Text: Towards Rigorous Benchmarking of Language Models (2022, background): “Though some work has studied how bias propagates downstream through language technologies [42, 59], an open challenge in designing benchmarks for language model harms is better understanding which metrics reﬂect harms in deployed use cases.”
A Survey on Gender Bias in Natural Language Processing (2021, methodology): “Kaneko and Bollegala [2019] split a given vocabulary into four mutually exclusive sets of word categories: words that are female-biased but non-discriminative, male-biased but non-discriminative, gender-neutral words, and words perpetuating stereotypes.”
Undesirable biases in NLP: Averting a crisis of measurement (2022, background): “…setting is especially prone to validity issues (Talat et al., 2022; Blodgett et al., 2021; Malik et al., 2022) and bias mitigation efforts do not necessarily transfer between languages even within the same multilingual model (Gonen et al., 2022) or between types of biases (Jin et al., 2021).”
Perturbation Augmentation for Fairer NLP (2022, background): “The empirical results are mixed, with some works finding that debiasing the pretrained model before finetuning does benefit down-stream tasks (Jin et al., 2021), but others find that intrinsic and extrinsic bias measurements do not correlate (Goldfarb-Tarrant et al., 2021; Cao et al., 2022),…”
On the Intrinsic and Extrinsic Fairness Evaluation Metrics for Contextualized Language Representations (2022, unknown): “While some prior work has demonstrated that biases in the up-stream language model have signiﬁcant effects on the downstream task fairness (Jin et al., 2021), others have shown that intrinsic and extrinsic metrics are not correlated (Goldfarb-Tarrant et al., 2021).”
Bias and Fairness in Large Language Models: A Survey (2023, background): “Other works have since followed this framework (Han et al., 2021b; Jin et al., 2021), training an encoder and discriminator, where the discriminator predicts a protected attribute from a”
Stepmothers are mean and academics are pretentious: What do pretrained language models learn about you? (2021, unknown): “These methods are, however, dependent on the dataset, domain, or task, making new mitigation needed when transferring to a new set-up (Jin et al., 2020).”
From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models (2023, background): “As extensive research has shown that machine learning models exhibit societal and political biases (Zhao et al., 2018; Blodgett et al., 2020b; Bender et al., 2021; Ghosh et al., 2021; Shaikh et al., 2022; Li et al., 2022; Cao et al., 2022; Goldfarb-Tarrant et al., 2021; Jin et al., 2021), there has ”