Fine-Grained Categorization and Dataset Bootstrapping Using Deep Metric Learning with Humans in the Loop

By Yin Cui, Feng Zhou, Yuanqing Lin, Serge J. Belongie. 2015

240 citations recorded by the source corpus.

Existing fine-grained visual categorization methods often suffer from three challenges: lack of training data, large number of fine-grained categories, and high intraclass vs. low inter-class variance. In this work we propose a generic iterative framework for fine-grained categorization and dataset bootstrapping that handles these three challenges. Using deep metric learning with humans in the loop, we learn a low dimensional feature embedding with anchor points on manifolds for each category. These anchor points capture intra-class variances and remain discriminative between classes. In each round, images with high confidence scores from our model are sent to humans for labeling. By comparing with exemplar images, labelers mark each candidate image as either a "true positive" or a "false positive." True positives are added into our current dataset and false positives are regarded as "hard negatives" for our metric learning model. Then the model is retrained with an expanded dataset and hard negatives for the next round. To demonstrate the effectiveness of the proposed framework, we bootstrap a fine-grained flower dataset with 620 categories from Instagram images. The proposed deep metric learning scheme is evaluated on both our dataset and the CUB-200-2001 Birds dataset. Experimental evaluations show significant performance gain using dataset bootstrapping and demonstrate state-of-the-art results achieved by the proposed deep metric learning methods.

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Explaining the Unexplained: A CLass-Enhanced Attentive Response (CLEAR) Approach to Understanding Deep Neural Networks (2017, background): “DNNs have been shown to outperform previous machine learning techniques for a variety of tasks, such as fine-grained classification [18, 5], self-driving cars [4], captioning and answering questions about images [9, 1], and even defeating human champions at Go [11].”
Large Scale Fine-Grained Categorization and Domain-Specific Transfer Learning (2018, methodology): “To capture subtle visual differences, visual attention [60, 16, 69] and deep metric learning [45, 10] are often used.”
Multi-Attention Multi-Class Constraint for Fine-grained Image Recognition (2018, methodology): “[6] uses the dataset bootstrapping with the help of human.”
Knowledge Concentration: Learning 100K Object Classifiers in a Single CNN (2017, background): “[4] proposed a generic iterative framework for fine-grained categorization and dataset bootstrapping which uses deep metric learning with humans in the loop.”
Online Deep Metric Learning (2018, background): “Existing DML algorithms can be broadly classified into two categories: (1) Convolutional neural networks (CNNs) combined with metric loss [5, 23, 27]; (2) Multilayer perceptron (MLP) (or Fully connected layers) combined with metric loss [10, 11].”
Attribute-Aware Attention Model for Fine-grained Representation Learning (2018, background): “ning projects images to a high dimensional embedding space, the main goal is to narrow the distances between similar images while pushing away dissimilar images as far as possible in embedding space. [8, 40] take image triplets as inputs and learn a similarity metric by adding triplet comparison con”
Imbalance Problems in Object Detection: A Review (2019, background): “[169] consider to exploit humans to label false positives during training, which are identified as hard examples and be added to the minibatch for the next iteration.”
Object-Part Attention Model for Fine-Grained Image Classification (2017, methodology): “30] 75.62 VGGNet Dense Graph Mining [38] 60.19 Coarse-to-Fine [39] p 82.50 VGGNet Coarse-to-Fine [39] p p 82.90 VGGNet PG Alignment [12] p p 82.80 VGGNet VGG-BGLm [13] p p 80.40 VGGNet Triplet-A (64) [40] p p 80.70 GoogleNet Triplet-M (64) [40] p p 79.30 GoogleNet Webly-supervised [41] p p 78.60 Ale”
Deep Metric Learning: A Survey (2019, methodology): “In [91], it was observed that some easy triplets had no effect in updating a model due to their poor discriminative power.”
Hard-Aware Deeply Cascaded Embedding (2016, background): “Deep metric learning attracts great attention in recent years, and hard negative mining is becoming a common practice to effectively train deep metric networks [8, 24, 39].”
A Metric Learning Reality Check (2020, background): “Pairs and triplets can also provide additional training signals for existing datasets [9].”
Adaptive Triplet Model for Fine-Grained Visual Categorization (2018, methodology): “In the fine-grained field, there are also some works [27], [31], [36] exploiting metric learning.”
Low-Rank Bilinear Pooling for Fine-Grained Classification (2016, background): “To alleviate the costly collection of part annotations, some have proposed to utilize interactive learning [6].”
Improved Deep Metric Learning with Multi-class N-pair Loss Objective (2016, background): “Despite their wide use, both loss functions are known to suffer from slow convergence and they often require expensive data sampling method to provide nontrivial pairs or triplets to accelerate the training [2, 19, 17, 4].”
Recognition From Web Data: A Progressive Filtering Approach (2018, unknown): “Similar collection methods are also used in works [12], [19], [35], [36], [42] which utilize web data for speciﬁc tasks.”
Learning a Discriminative Filter Bank Within a CNN for Fine-Grained Recognition (2016, background): “Others have explored directions such as utilizing hierarchical label structures [57], combining visual and textual information [54, 2, 16], 3D-assisted recognition [26, 31, 37], introducing humans in the loop [40, 4, 7], and collecting larger amount of data [45, 47, 25, 17].”
Improving Fine-Grained Visual Classification using Pairwise Confusion (2017, methodology): “Similar to LSVC tasks, neural networks for FGVC tasks are typically trained with cross-entropy loss [1,7,8,9].”