Long-Tailed Classification by Keeping the Good and Removing the Bad Momentum Causal Effect

By Kaihua Tang, Jianqiang Huang, Hanwang Zhang. 2020

525 citations recorded by the source corpus.

As the class size grows, maintaining a balanced dataset across many classes is challenging because the data are long-tailed in nature; it is even impossible when the sample-of-interest co-exists with each other in one collectable unit, e.g., multiple visual instances in one image. Therefore, long-tailed classification is the key to deep learning at scale. However, existing methods are mainly based on re-weighting/re-sampling heuristics that lack a fundamental theory. In this paper, we establish a causal inference framework, which not only unravels the whys of previous methods, but also derives a new principled solution. Specifically, our theory shows that the SGD momentum is essentially a confounder in long-tailed classification. On one hand, it has a harmful causal effect that misleads the tail prediction biased towards the head. On the other hand, its induced mediation also benefits the representation learning and head prediction. Our framework elegantly disentangles the paradoxical effects of the momentum, by pursuing the direct causal effect caused by an input sample. In particular, we use causal intervention in training, and counterfactual reasoning in inference, to remove the "bad" while keep the "good". We achieve new state-of-the-arts on three long-tailed visual recognition benchmarks: Long-tailed CIFAR-10/-100, ImageNet-LT for image classification and LVIS for instance segmentation.

Subfields: Artificial Intelligence, Causal Inference & Discovery

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Causal Inference & Discovery

Citations in the corpus

Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation (2020, background): “Other more complicated learning methods ( e.g ., meta-learning [63, 29, 32], causal inference [58], Bayesian methods [34], etc .) are also used to deal with long-tail data.”
Distilling Causal Effect of Data in Class-Incremental Learning (2021, methodology): “It has been recently introduced to various computer vision fields, including feature learning [3, 46], few-shot classification [49], long-tailed recognition [40], semantic segmentation [8] and other high-level vision tasks[41, 32, 48, 27].”
Interventional Few-Shot Learning (2020, background): “[67] proposes a retrospective for image captioning and other applications include image classification [10, 36], imitation learning [13], long-tailed recognition [56] and semantic segmentation [73].”
Delving into Deep Imbalanced Regression (2021, unknown): “This phenomenon poses great challenges for deep recognition models, and has motivated many prior techniques for addressing data imbalance (Cao et al., 2019; Cui et al., 2019; Huang et al., 2019; Liu et al., 2019; Tang et al., 2020).”
CReST: A Class-Rebalancing Self-Training Framework for Imbalanced Semi-Supervised Learning (2021, background): “A recent trend of work proposes to decouple the learning of representation and classiﬁer [52, 20, 42].”
Conterfactual Generative Zero-Shot Semantic Segmentation (2021, background): “Specifically, in the computer vision field, the defect of pure statistical relationship may worsen dataset bias [17].”
Deep Long-Tailed Learning: A Survey (2021, methodology): “Causal classifier [88] resorted to causal inference for keeping the good and removing the bad momentum causal effects in longtailed learning.”
BENDR: Using Transformers and a Contrastive Self-Supervised Learning Task to Learn From Massive Amounts of EEG Data (2021, background): “This would no doubt be imbalanced; the labels would be distributed in a long-tailed or Zipfian distribution that would likely require well thought-out adjustment [68, 69].”
Distribution Alignment: A Unified Framework for Long-tail Visual Recognition (2021, methodology): “[28, 35] improve the two-stage ideas by introducing a post-process to adjust the prediction score.”
Learning to Imagine: Integrating Counterfactual Thinking in Neural Discrete Reasoning (2022, background): “Another line of research performs counterfactual inference over the predictions of deep model to incorporate counterfactual thinking (Yue et al., 2021; Wang et al., 2021; Niu et al., 2021; Tang et al., 2020).”
Causal Intervention for Weakly-Supervised Semantic Segmentation (2020, background): “Recently, there is a growing number of computer vision tasks that benefit from causality [34, 40, 51, 52, 56, 63, 65].”