Large-Scale Long-Tailed Recognition in an Open World

By Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, Stella X. Yu. 2019

1,434 citations recorded by the source corpus.

Real world data often have a long-tailed and open-ended distribution. A practical recognition system must classify among majority and minority classes, generalize from a few known instances, and acknowledge novelty upon a never seen instance. We define Open Long-Tailed Recognition (OLTR) as learning from such naturally distributed data and optimizing the classification accuracy over a balanced test set which include head, tail, and open classes. OLTR must handle imbalanced classification, few-shot learning, and open-set recognition in one integrated algorithm, whereas existing classification approaches focus only on one aspect and deliver poorly over the entire class spectrum. The key challenges are how to share visual knowledge between head and tail classes and how to reduce confusion between tail and open classes. We develop an integrated OLTR algorithm that maps an image to a feature space such that visual concepts can easily relate to each other based on a learned metric that respects the closed-world classification while acknowledging the novelty of the open world. Our so-called dynamic meta-embedding combines a direct image feature and an associated memory feature, with the feature norm indicating the familiarity to known classes. On three large-scale OLTR datasets we curate from object-centric ImageNet, scene-centric Places, and face-centric MS1M data, our method consistently outperforms the state-of-the-art. Our code, datasets, and models enable future OLTR research and are publicly available at \url{https://liuziwei7.github.io/projects/LongTail.html}.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

In Evaluation & Benchmarks

Citations in the corpus

Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts (2021, background): “Long-tail Visual Recognition in V+L Addressing long-tail distributions of visual concepts is an important component of V+L systems that generalize, as long and free-form texts exhibit a large number of compositional, ﬁne-grained categories [84, 51, 19].”
Open-World Instance Segmentation: Exploiting Pseudo Ground Truth From Learned Pairwise Affinity (2022, neutral): “Open-world setup [5, 43] has been recently (re)-introduced into various problems in computer vision such as recognition [30, 38], tracking [36], detection [25, 27, 53], and segmentation [54].”
Towards fairer datasets: filtering and balancing the distribution of the people subtree in the ImageNet hierarchy (2019, background): “First, the accuracy of a machine learning model is strongly influenced by the number of training examples: underrepresented categories in datasets will be inherently more challenging for the model to learn [51].”
Long-Tailed Class Incremental Learning (2022, background): “However, data sampled from the real world often follows a long-tailed distribution [14,24,40,49] in which some classes have many more samples than others.”
Towards Open World Object Detection (2021, background): “[35] considered a long-tailed recognition setting where majority, minority and unknown classes coexist.”
Imbalanced Continual Learning with Partitioning Reservoir Sampling (2020, methodology): “This is a common practice in imbalanced classification benchmarks, including [85] that uses 40 test images per class in the SUN-LT dataset and the largest OLTR benchmark [55] that assigns 50 and 100 balanced test images per class for ImageNet-LT and Places-LT dataset, respectively.”
Incremental Few-Shot Object Detection (2020, neutral): “Such a missing capability is required in robotics applications [32, 1], when the detector is running on embedded devices, or simply to scale up to addressing the long tail of object categories to recognise [31].”
Decoupling Representation and Classifier for Long-Tailed Recognition (2019, methodology): “As we see in Table 4, all three decoupled methods outperform the state-of-the-art approaches, including Lifted Loss (Oh Song et al., 2016), Focal Loss (Lin et al., 2017), Range Loss (Zhang et al., 2017), FSLwF (Gidaris & Komodakis, 2018) and OLTR (Liu et al., 2019).”
Learning Adaptive Classifiers Synthesis for Generalized Few-Shot Learning (2019, background): “manner [16, 21, 40, 66, 70].”
Unidentified Video Objects: A Benchmark for Dense, Open-World Segmentation (2021, background): “Openworld problems have been studied in the context of recognition [3, 25]: given a closed-world training dataset, how to actively identify new object categories? To handle novel objects, previous works explicitly differentiate unknown from known, such as by spotting outliers in the embedding space.”
Recent Advances in Open Set Recognition: A Survey (2018, background): “Further, as the object frequencies in visual categories ordinarily follow long-tailed distribution [29], [117], such a distribution fitting will face challenges once the rare classes in KKCs and UUCs appear together in testing [118].”
Exploring Balanced Feature Spaces for Representation Learning (2021, methodology): “To thoroughly investigate sensitiveness of different representation learning methods to the imbalance level of training datasets, we construct six datasets from the long-tailed benchmark ImageNetLT (Liu et al., 2019) (DLT) by varying its instance distribution {q1, .”
Distribution Alignment: A Unified Framework for Long-tail Visual Recognition (2021, methodology): “Early works on re-balancing data distribution focus on learning one-stage models, which achieve limited successes due to lack of principled design in their strategies [2, 33, 3, 9, 26, 41].”
Is Label Smoothing Truly Incompatible with Knowledge Distillation: An Empirical Study (2021, methodology): “Place-LT (Liu et al., 2019).”
ELF: An Early-Exiting Framework for Long-Tailed Classification (2020, methodology): “We conduct our evaluation on four long-tailed datasets: CIFAR-10 LT, CIFAR-100 LT [6], ImageNet LT [12] and iNaturalist’18 [35].”
Solving Long-tailed Recognition with Deep Realistic Taxonomic Classifier (2020, methodology): “ImageNet-LT [46] is a long-tailed version of [16], with 1000 classes of more than 5 and less than 1280 images per class, and a balanced test set. iNaturalist (2018) [1,33] is a large-scale dataset of 8 142 classes with the class imbalance factor of 0 .”
A Spectral-Spatial-Dependent Global Learning Framework for Insufficient and Imbalanced Hyperspectral Image Classification (2021, background): “Existing solutions usually have a difference in sampling strategy and classifiers [44].”
Equalization Loss for Long-Tailed Object Recognition (2020, methodology): “We use ResNet-10 as our backbone like [30].”
Opening up Open World Tracking (2021, background): “By contrast, open-world recognition methods, as defined by [5, 43] must explicitly recognize unknown object instances that were not observed during training, and update object detectors to recognize these unknown instances.”