The Devil is in the Tails: Fine-grained Classification in the Wild

By Grant Van Horn, Pietro Perona. 2017

224 citations recorded by the source corpus.

The world is long-tailed. What does this mean for computer vision and visual recognition? The main two implications are (1) the number of categories we need to consider in applications can be very large, and (2) the number of training examples for most categories can be very small. Current visual recognition algorithms have achieved excellent classification accuracy. However, they require many training examples to reach peak performance, which suggests that long-tailed distributions will not be dealt with well. We analyze this question in the context of eBird, a large fine-grained classification dataset, and a state-of-the-art deep network classification algorithm. We find that (a) peak classification performance on well-represented categories is excellent, (b) given enough data, classification performance suffers only minimally from an increase in the number of classes, (c) classification performance decays precipitously as the number of training examples decreases, (d) surprisingly, transfer learning is virtually absent in current methods. Our findings suggest that our community should come to grips with the question of long tails.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

Self-Supervised Learning of Pretext-Invariant Representations (2019, background): “Pre-defined semantic annotations scale poorly to the long tail of visual concepts [75], which hampers further improvements in image recognition.”
Federated Visual Classification with Real-World Data Distribution (2020, background): “While studying learning robustly with differing training and test distributions is a topic for research [29] in itself, in our federated learning benchmark, we create class-balanced training and test sets with uniform distributions.”
Recognition in Terra Incognita (2018, background): “Furthermore, the number of images per class is balanced, unlike what happens in the real world [11].”
Imbalanced Continual Learning with Partitioning Reservoir Sampling (2020, background): “This naturally occurring phenomenon is widely observed in vision and language datasets [70,63,80], with a whole other branch of machine learning that has focused solely on this topic.”
Incremental Meta-Learning via Indirect Discriminant Alignment (2020, background): “Deep neural networks trained for classification [18, 12, 14] do not fare well when trained with small datasets typical of the tail [37], leading to increased interest in few-shot learning.”
What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation (2020, background): “While it has been recognized in many contexts that modern datasets are long-tailed [32, 28, 3], it is unclear whether this has any relationship to memorization by modern deep learning algorithms and (if so) how significant is the effect.”
Does learning require memorization? a short tale about a long tail (2019, background): “different conditions (such as close-ups, in foliage and in the sky) [49].”
Class-Balanced Loss Based on Effective Number of Samples (2019, background): “CNNs trained on such data perform poorly for weakly represented classes [19, 15, 41, 4].”
Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss (2019, methodology): “Modern real-world large-scale datasets often have long-tailed label distributions [51, 28, 34, 12, 15, 50, 40].”
BBN: Bilateral-Branch Network With Cumulative Learning for Long-Tailed Visual Recognition (2019, background): “While, the re-balancing branch aims to alleviate the extreme imbalance and particularly improve the classification accuracy on tail classes [26], whose input data comes from a reversed sampler.”
Learning to Model the Tail (2017, background): “pre-training on the head is quite similar to training on the unbalanced long-tailed dataset (which is dominated by the head) [10].”
Separating Self-Expression and Visual Content in Hashtag Supervision (2017, background): “An important obstacle to increasing variety is that successful recognition of the long tail of visual content [11] may require manual annotation of hundreds of millions of images into hundreds of thousands of classes, which is difﬁcult and time-consuming.”
Task-Driven Modular Networks for Zero-Shot Compositional Learning (2019, background): “observed that visual concepts follow a long-tailed distribution [28, 34, 31].”
Balanced Knowledge Distillation for Long-tailed Learning (2021, background): “However, in practical scenarios, data tends to exhibit long-tailed distribution [38, 17], wherein a few classes (head classes) have a significantly larger number of instances than other classes (tail classes).”