Learning Not to Learn: Training Deep Neural Networks With Biased Data

By Byungju Kim, Hyunwoo Kim, Kyungsu Kim, Sungjin Kim, Junmo Kim. 2018

468 citations recorded by the source corpus.

We propose a novel regularization algorithm to train deep neural networks, in which data at training time is severely biased. Since a neural network efficiently learns data distribution, a network is likely to learn the bias information to categorize input data. It leads to poor performance at test time, if the bias is, in fact, irrelevant to the categorization. In this paper, we formulate a regularization loss based on mutual information between feature embedding and bias. Based on the idea of minimizing this mutual information, we propose an iterative algorithm to unlearn the bias information. We employ an additional network to predict the bias distribution and train the network adversarially against the feature embedding network. At the end of learning, the bias prediction network is not able to predict the bias not because it is poorly trained, but because the feature embedding network successfully unlearns the bias information. We also demonstrate quantitative and qualitative experimental results which show that our algorithm effectively removes the bias information from feature embedding.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

DISCO: Dynamic and Invariant Sensitive Channel Obfuscation for deep neural networks (2020, methodology): “Baselines For attacks on sensitive attributes, we baseline with ARL based methods [29, 24, 23] which are state-ofthe-art on attribute leakage.”
Don’t Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases (2019, background): “A popular approach is to use an adversary to remove information about a target feature, often gender or ethnicity, from a model’s internal representations (Edwards and Storkey, 2016; Wang et al., 2018; Kim et al., 2019).”
Explaining Classifiers with Causal Concept Effect (CaCE) (2019, methodology): “The colored-MNIST dataset (Kim et al., 2019) is a variant of the MNIST dataset (Deng, 2012) where the foreground digit is colored.”
Domain Generalization: A Survey (2021, background): “For example, given the task of distinguishing between digits blended with different colors, a neural network might be biased toward recognizing colors rather than the digit shapes during training, thus leading to poor generalization on unseen data [242].”
Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond (2021, background): “…of the threats to generalization posed by spurious correlations, 2A prominent example is “colored MNIST”, in which the color channel is utilized to create challenging distributional shifts (Kim et al., 2019). distributional shift, and causally-motivated approaches to achieving greater robustness.”
Towards Fairness in Visual Recognition: Effective Strategies for Bias Mitigation (2019, background): “Classical literature on mitigating bias generally operates on simpler (often linear) models [11, 50, 28], which are easier to understand and control; only recently have researchers begun looking at mitigating bias in end-to-end trained deep learning models [16, 2, 40, 18, 48, 25, 30, 36, 47, 17].”
Why Can't I Dance in the Mall? Learning to Mitigate Scene Bias in Action Recognition (2019, background): “Examples include reducing gender or age biases for fairness [2, 5, 33, 58], easy-example biases [59, 49], and texture bias of image CNNs for improved generalization [17].”
Contextualizing Multiple Tasks via Learning to Decompose (2021, result): “Similar to [3, 16, 23], which plot RGB colors over the grayscale images based on the Kuzushiji-49 dataset [9] and test whether a model will be disturbed by the colors.”
Network Randomization: A Simple Technique for Generalization in Deep Reinforcement Learning (2019, methodology): “5 Following the same setup as Kim et al. (2019), we construct datasets with an undesirable bias as follows: the training set consists of bright dogs and dark cats while the test set consists of dark dogs and bright cats (see Appendix H for further details).”
Counterfactual Generative Networks (2021, methodology): “We compare to current approaches for training invariant classifiers: IRM (Arjovsky et al., 2019) and Learning-not-to-learn (LNTL) (Kim et al., 2019).”
Right for the Right Concept: Revising Neuro-Symbolic Concepts by Interacting with their Explanations (2020, background): “ToyColor [43] 40k 2 5× 5× 3 ✗ ✓ ✗ 1 ColorMNIST [17] 70k 10 28× 28× 3 ✗ ✗ ✓ 1 Decoy-MNIST [43] 70k 10 28× 28× 3 ✗ ✓ ✗ 1 Plant Data Set [44] 2.”
Machine learning and deep learning—A review for ecologists (2022, unknown): “…should not interpret xAI metrics as “effect estimates”— if a certain feature is strongly used by the ML model, it could be because this feature has a biological or ecological effect on the response, but it could also easily be because it correlates with other features (e.g. Genuer et al., 2010 ).”
Learning De-biased Representations with Biased Representations (2019, unknown): “, 2019b) or mutual information minimisation (Kim et al., 2019; Creager et al., 2019).”
The Medium Is Not the Message: Deconfounding Document Embeddings via Linear Concept Erasure (2025, unknown): “Other works have developed unsupervised techniques to debias or erase neural representations (Kim et al., 2019; Seo et al., 2022; Yang et al., 2025).”