Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection

By Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, Yoav Goldberg. 2020

555 citations recorded by the source corpus.

The ability to control for the kinds of information encoded in neural representation has a variety of use cases, especially in light of the challenge of interpreting these models. We present Iterative Null-space Projection (INLP), a novel method for removing information from neural representations. Our method is based on repeated training of linear classifiers that predict a certain property we aim to remove, followed by projection of the representations on their null-space. By doing so, the classifiers become oblivious to that target property, making it hard to linearly separate the data according to it. While applicable for multiple uses, we evaluate our method on bias and fairness use-cases, and show that our method is able to mitigate bias in word embeddings, as well as to increase fairness in a setting of multi-class classification.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Language (Technology) is Power: A Critical Survey of “Bias” in NLP (2020, neutral): “…(2019); Vaidya et al. (2019); Zhiltsova et al. (2019); Chopra et al. (2020); Gonen and Webster (2020); Gyamfi et al. (2020); Hube et al. (2020); Ravfogel et al. (2020); Rios (2020); Ross et al. (2020); Saunders and Byrne (2020); Sen and Ganguly (2020); Shah et al. (2020); Sweeney and Najafian…”
Harms of Gender Exclusivity and Challenges in Non-Binary Representation in Language Technologies (2021, neutral): “…language model biases such as gender, race or religion-related stereotypes in static word embeddings (GloVe (Pennington et al., 2014)) and contextual (e.g., BERT (Devlin et al., 2019)) representations (Bolukbasi et al., 2016; De-Arteaga et al., 2019; Ravfogel et al., 2020; Dev et al., 2020b).”
Towards Understanding and Mitigating Social Biases in Language Models (2021, methodology): “, 2019) and mitigating (Liang et al., 2020; Ravfogel et al., 2020) bias in contextual embeddings such as ELMo (Peters et al.”
Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals (2020, neutral): “Voita and Titov (2020) follow this direction, using the concept of minimum description length (MDL, Rissanen, 1978) to quantify the total information needed to transmit both the probing model and the labels it predicts.”
CausaLM: Causal Model Explanation Through Counterfactual Language Models (2020, methodology): “The corpus-based estimator of CONEXP is defined as follows: CONEXPC0(O) = 1 I [∑ i∈I 〈~z(f(φO(xi)|Cj = 1)− ~z(f(φO(xi)|Cj = 0)〉 ] (13) The second baseline we consider is TPR-GAP, introduced in De-Arteaga et al. (2019) and used by Ravfogel et al. (2020).”
Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP (2021, methodology): “For static word embeddings, various algorithms for debiasing have been proposed (Bolukbasi et al., 2016; Zhao et al., 2018; Ravfogel et al., 2020; Go-nen and Goldberg, 2019), many of them being based on predeﬁned word lists or other external resources.”
Multi-Dimensional Gender Bias Classification (2020, background): “moving gender bias from text (Emami et al., 2019; Hall Maudslay et al., 2019; Ravfogel et al., 2020), no extant work on classifying gender or removing gender bias has foregrounded the fact that we use language, at least in part, to collaboratively and socially construct our gender identities.”
UNQOVERing Stereotypical Biases via Underspecified Questions (2020, background): “Such studies on model bias have led to many bias mitigation techniques (e.g., Bolukbasi et al., 2016b; Dev et al., 2020a; Ravfogel et al., 2020; Dev et al., 2020b).”
Formalizing Trust in Artificial Intelligence: Prerequisites, Causes and Goals of Human Trust in AI (2020, background): “governance Ensure privacy and data protection Datasheets/statements Removal of protected attributes [57] Ensure quality and integrity of data Datasheets/statements Detecting data artifacts [23]”
Bias and Fairness in Large Language Models: A Survey (2023, methodology): “Adapting INLP to a non-linear classifer, Iskander et al. (2023) proposes Iterative Gradient-Based Projection (IGBP), which leverages the gradients of a neural protected attribute classifier to project representations to the classifier’s class boundary, which should make the representations indisting”
Theories of “Gender” in NLP Bias Research (2022, background): “664737/FULL [115] Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg.”
The Geometry of Distributed Representations for Better Alignment, Attenuated Bias, and Improved Interpretability (2020, methodology): “It is thus, not sufficient to claim using this that bias has been removed or that valid gender associations have been retained, leading to the requirement of methods described in the paper and methods in other work [12, 17, 28, 76] for the same.”
An Empirical Survey of the Effectiveness of Debiasing Techniques for Pre-trained Language Models (2021, methodology): “Because of these ﬁndings, an increasing amount of research has focused on developing techniques to mitigate these biases (Liang et al., 2020; Ravfogel et al., 2020; Webster et al., 2020; Kaneko and Bollegala, 2021; Schick et al., 2021; Lauscher et al., 2021).”
Mechanistic Interpretability for AI Safety - A Review (2024, methodology): “…et al., 2023), unifying various interpretability methods such as LIME (Ribeiro et al., 2016), causal effect estimation (Feder et al., 2021), causal mediation analysis (Vig et al., 2020), iterated nullspace projection (Ravfogel et al., 2020), and circuit-based explanations (Geiger et al., 2023a).”
StylePTB: A Compositional Benchmark for Fine-grained Controllable Text Style Transfer (2021, background): “Unconditional language models have been shown to perpetuate undesirable stereotypes during generation which disproportionately harm underrepresented social groups (Liang et al., 2020b; Ravfogel et al., 2020; Sheng et al., 2020).”
OSCaR: Orthogonal Subspace Correction and Rectification of Biases in Word Embeddings (2020, methodology): “It then automatically identifies a second set B of most biased words: these are the most extreme words along the direction v (or −v) [16].”
Benchmarking Intersectional Biases in NLP (2022, methodology): “We debiased the GloVe model using (Ravfogel et al., 2020).”
Teach Me to Explain: A Review of Datasets for Explainable NLP (2021, background): “…consider pruning the explanation template type from intermediate representations of a base model, e.g., with adversarial training (Goodfellow et al., 2015) or more sophisticated methods (Ravfogel et al., 2020), to avoid models learning label-speciﬁc templates rather than explanation semantics.”
Refusal in Language Models Is Mediated by a Single Direction (2024, unknown): “The assumption that features are represented linearly has been effective for erasing concepts from language models (Belrose, 2023; Belrose et al., 2024; Guerner et al., 2023; Haghighatkhah et al., 2022; Ravfogel et al., 2020; Shao et al., 2022).”