Long-tail learning via logit adjustment

By Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, Sanjiv Kumar. International Conference on Learning Representations, 2020

1,020 citations recorded by the source corpus.

Real-world classification problems typically exhibit an imbalanced or long-tailed label distribution, wherein many labels are associated with only a few samples. This poses a challenge for generalisation on such labels, and also makes naive learning biased towards dominant labels. In this paper, we present two simple modifications of standard softmax cross-entropy training to cope with these challenges. Our techniques revisit the classic idea of logit adjustment based on the label frequencies, either applied post-hoc to a trained model, or enforced in the loss during training. Such adjustment encourages a large relative margin between logits of rare versus dominant labels. These techniques unify and generalise several recent proposals in the literature, while possessing firmer statistical grounding and empirical performance.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/4f65f604d3bf5fa91634484a3232b426267b71ef

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Wen Li
2. Lixin Duan
3. Yahao Liu
4. Jinhong Deng
5. Tong Chu
6. Luc Van Gool
7. Jian‐Yuan Sun
8. Kihyuk Sohn

Citations in the corpus

• Distribution Alignment: A Unified Framework for Long-tail Visual Recognition (2021, methodology): “Notably, Logit Adjustment[28] and Deconfound[35] introduce a hand-craft Method Align Method”
• Contrastive Learning based Hybrid Networks for Long-Tailed Image Classification (2021, background): “An intuitive alternative is to shift the separator closer to a dominant class [27].”
• Graph Few-shot Class-incremental Learning (2021, methodology): “Specifically, our framework uses a dynamically scaled cross-entropy loss regularizer where the scale factors[20, 24] are multiplied to each task-level loss to adjust their contribution for model training.”
• CReST: A Class-Rebalancing Self-Training Framework for Imbalanced Semi-Supervised Learning (2021, methodology): “3.4 and Logit Adjustment (LA) [31], an ad-hoc post-processing technique to enhance models’ discriminative ability on minority classes by adjusting the logits of model predictions.”
• Balanced Knowledge Distillation for Long-tailed Learning (2021, methodology): “(8) BBN: BilateralBranch Network [56]; (9) cRT: first learning representation and then re-training classifier [21]; (10) Logit Adjustment loss: the logit adjusted softmax cross-entropy loss [36].”
• Undoing the Damage of Label Shift for Cross-domain Semantic Segmentation (2022, background): “As stated in [4, 33], while re-weigting the loss with the label distribution ratio also helps to guide the classifier to approach the target classifier, yet not as effectively as directly using it to correct the classifier output like our classifier refinement loss.”