Decoupling Representation and Classifier for Long-Tailed Recognition

By Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, Yannis Kalantidis. 2019

1,567 citations recorded by the source corpus.

The long-tail distribution of the visual world poses great challenges for deep learning based classification models on how to handle the class imbalance problem. Existing solutions usually involve class-balancing strategies, e.g., by loss re-weighting, data re-sampling, or transfer learning from head- to tail-classes, but most of them adhere to the scheme of jointly learning representations and classifiers. In this work, we decouple the learning procedure into representation learning and classification, and systematically explore how different balancing strategies affect them for long-tailed recognition. The findings are surprising: (1) data imbalance might not be an issue in learning high-quality representations; (2) with representations learned with the simplest instance-balanced (natural) sampling, it is also possible to achieve strong long-tailed recognition ability by adjusting only the classifier. We conduct extensive experiments and set new state-of-the-art performance on common long-tailed benchmarks like ImageNet-LT, Places-LT and iNaturalist, showing that it is possible to outperform carefully designed losses, sampling strategies, even complex modules with memory, by using a straightforward approach that decouples representation and classification. Our code is available at this https URL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation (2020, methodology): “Recent work [9, 3, 33, 71, 37] has pointed out the effectiveness of two-stage training strategies by separating the feature learning and the re-balancing stage, as end-to-end training with re-balancing strategies could be detrimental to feature learning.”
Few-Shot Class-Incremental Learning from an Open-Set Perspective (2022, neutral): “As real-world classification problems typically exhibit class imbalance or long-tailed data distribution, some methods have explored deep metric learning for incremental and long-tail tasks [22,15,12].”
Overcoming Catastrophic Forgetting in Incremental Few-Shot Learning by Finding Flat Minima (2021, methodology): “Method sessions The gap with cRT 1 2 3 4 5 6 7 8 9 cRT [25]∗ 65.”
DER: Dynamically Expandable Representation for Class Incremental Learning (2021, methodology): “To this end, we adopt a two-stage learning strategy, decoupling the adaptation of feature representation and final classifier head (or classifier for short) of a deep network [15].”
Learning Adaptive Classifiers Synthesis for Generalized Few-Shot Learning (2019, background): “Directly training a system over all classes is prone to over-fit and can be biased towards the data-rich categories [6, 12, 25, 69, 72].”
Rethinking the Value of Labels for Improving Class-Imbalanced Learning (2020, methodology): “We again evaluate models on the corresponding balanced test datasets [7,25,33].”
FedBABU: Towards Enhanced Representation for Federated Image Classification (2021, methodology): “Inspired by prior studies on long-tailed recognition (Yu et al., 2020; Kang et al., 2019), fine-tuning (Devlin et al., 2019), and self-supervised learning (He et al., 2020), as well as our data-sharing experiment, we decouple the entire network into the body and the head.”
Remix: Rebalanced Mixup (2020, methodology): “Other Types Two competitive state-of-the arts [38,17] focus on the representation learning and classifier learning of a CNN.”
CReST: A Class-Rebalancing Self-Training Framework for Imbalanced Semi-Supervised Learning (2021, background): “Various solutions have been proposed to help alleviate bias, such as re-sampling [3, 4], re-* weighting [9, 6], and two-stage training [20, 52].”
Language-Grounded Indoor 3D Semantic Segmentation in the Wild (2022, background): “…towards data-sampling-based re-balancing, for instance random oversampling of underrepresented classes [3,54,50], sampling novel poses of known categories [32], undersampling overrepresented classes [35], frequency-based sampling [25], as well as feature-based or generative sampling [40,39,55].”
On the Soft-Subnetwork for Few-shot Class Incremental Learning (2022, methodology): “Furthermore, we fix the classifier re-training method (cRT) [22] for long-tailed classification trained with all encountered data as the approximated upper bound.”
Disentangling Label Distribution for Long-tailed Visual Recognition (2020, methodology): “Rows with † denote results directly borrowed from [28].”
Long-Tailed Classification by Keeping the Good and Removing the Bad Momentum Causal Effect (2020, neutral): “The sharing bilateral-branch network [10], the two-stage training [11], the dynamic curriculum learning [33] and the transferring memory features [9] / head distributions [34] are all shown to be effective in long-tailed recognition, yet, they either significantly increase the parameters or require…”
Discriminability-Transferability Trade-Off: An Information-Theoretic Perspective (2022, background): “Recent works [18,48] revealed that the performance of a classification model is closely related to the quality of its classifier.”
Distribution-Balanced Loss for Multi-Label Classification in Long-Tailed Datasets (2020, background): “Previous works on long-tailed recognition [34,27,18] mainly follow two directions: re-sampling and cost-sensitive learning.”
The Devil is in Classification: A Simple Framework for Long-tail Instance Segmentation (2020, background): “In addition, [23] tries to examine the relation of feature and classifier learning in an imbalanced setting.”