Adversarial Removal of Demographic Attributes Revisited

By Maria Barrett, Yova Kementchedjhieva, Yanai Elazar, Desmond Elliott, Anders Søgaard. 2019

63 citations recorded by the source corpus.

Elazar and Goldberg (2018) showed that protected attributes can be extracted from the representations of a debiased neural network for mention detection at above-chance levels, by evaluating a diagnostic classifier on a held-out subsample of the data it was trained on. We revisit their experiments and conduct a series of follow-up experiments showing that, in fact, the diagnostic classifier generalizes poorly to both new in-domain samples and new domains, indicating that it relies on correlations specific to their particular data sample. We further show that a diagnostic classifier trained on the biased baseline neural network also does not generalize to new samples. In other words, the biases detected in Elazar and Goldberg (2018) seem restricted to their particular data sample, and would therefore not bias the decisions of the model on new samples, whether in-domain or out-of-domain. In light of this, we discuss better methodologies for detecting bias in our models.

Subfields: Artificial Intelligence, General Social Sciences, AI Ethics, Fairness & Society

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In General Social Sciences

In AI Ethics, Fairness & Society

Citations in the corpus

Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias (2020, background): “Barrett et al. (2019) further demonstrate that classiﬁers that aim to detect biases in learned representations focus on spurious correlations in their training data and fail to generalize to unseen data.”
Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection (2020, background): “While adverserial methods showed impressive performance in various machine learning tasks, and were applied for the goal of removal of sensitive information (Elazar and Goldberg, 2018; Coavoux et al., 2018; Resheff et al., 2019; Barrett et al., 2019), they are notoriously hard to train.”
Measuring and Reducing Gendered Correlations in Pre-trained Models (2020, neutral): “In particular, Gonen and Goldberg (2019) found that gender information is still retrievable after having applied adversarial removal, while Barrett et al. (2019) presented a follow up study showing that such results only hold when models are deployed on the same data on which they were trained.”
Investigating Gender Bias in Language Models Using Causal Mediation Analysis (2020, background): “Moreover, Barrett et al. (2019) showed that probing classiﬁers may generate unfaithful interpretations and fail to generalize to unseen data.”