Deep Metric Learning via Lifted Structured Feature Embedding

By Hyun Oh Song, Yu Xiang, Stefanie Sabrina Jegelka, Silvio Savarese. 2015

1,791 citations recorded by the source corpus.

Learning the distance metric between pairs of examples is of great importance for learning and visual recognition. With the remarkable success from the state of the art convolutional neural networks, recent works [1, 31] have shown promising results on discriminatively training the networks to learn semantic feature embeddings where similar examples are mapped close to each other and dissimilar examples are mapped farther apart. In this paper, we describe an algorithm for taking full advantage of the training batches in the neural network training by lifting the vector of pairwise distances within the batch to the matrix of pairwise distances. This step enables the algorithm to learn the state of the art feature embedding by optimizing a novel structured prediction objective on the lifted problem. Additionally, we collected Stanford Online Products dataset: 120k images of 23k classes of online products for metric learning. Our experiments on the CUB-200-2011 [37], CARS196 [19], and Stanford Online Products datasets demonstrate significant improvement over existing deep feature embedding methods on all experimented embedding sizes with the GoogLeNet [33] network. The source code and the dataset are available at: https://github.com/rksltnl/ Deep-Metric-Learning-CVPR16.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

In Representation & Self-Supervised Learning

Citations in the corpus

Revisiting IM2GPS in the Deep Learning Era (2017, methodology): “New loss functions and example mining strategies have been being proposed as they play important role in the learning process [24, 20, 23, 34].”
Unsupervised Embedding Learning via Invariant and Spreading Instance Feature (2019, methodology): “Deep embedding learning is a fundamental task in computer vision [14], which aims at learning a feature embedding that has the following properties: 1) positive concentrated, the embedding features of samples belonging to the same category are close to each other [32]; 2) negative separated, the emb”
Time-Contrastive Networks: Self-Supervised Learning from Video (2017, neutral): “We also experiment with other metric learning losses, namely npairs [42] and lifted structured [43], and show that results are comparable.”
Occupancy Networks: Learning 3D Reconstruction in Function Space (2018, methodology): “While we train and test our method on the ShapeNet dataset, we also present qualitative results for the KITTI [22] and the Online Products dataset [49].”
Learning Spread-Out Local Feature Descriptors (2017, methodology): “In the experiment section, we test combining the global orthogonal regularization with contractive loss3 (1) [19], triplet loss (2) [1], lifted structured similarity softmax loss (LSSS) [14] and N-pair loss [21].”
Continual Prototype Evolution: Learning Online from Non-Stationary Data Streams (2020, neutral): “losses, whereas other works use batch information in lifted structure embeddings [35] or instance-wise softmax embeddings [51].”
VoxCeleb2: Deep Speaker Recognition (2018, methodology): “We use an ofﬂine hard negative mining strategy, which allows us to select harder negatives ( e.g. top 1-percent of randomly generated pairs) than is possible with online (in-batch) hard negative mining methods [42, 41, 43] limited by the batch size.”
SphereFace: Deep Hypersphere Embedding for Face Recognition (2017, background): “Recently, prevailing deep metric learning [7, 17, 24, 30, 25, 22, 34] usually uses neural networks to automatically learn discriminative features x1,x2 followed by a simple distance metric such as Euclidean distance ‖x1−x2‖2.”
3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction (2016, methodology): “We further tested the network’s ability to perform multi-view reconstruction on the ShapeNet dataset [1] and the Online Products dataset [46], which showed that the network is able to incrementally improve its reconstructions as it sees more views of an object.”
Relational Knowledge Distillation (2019, methodology): “As embedding models are commonly evaluated on image retrieval, we validate our approach using image retrieval benchmarks of CUB-200-2011 [40], Cars 196 [14], and Stanford Online Products [21] datasets and we follow the train/test splits suggested in [21].”
Large-Scale Long-Tailed Recognition in an Open World (2019, methodology): “We choose for comparison stateof-the-art methods from different fields dealing with the open long-tailed data, including: (1) metric learning: Lifted Loss [33], (2) hard negative mining: Focal Loss [27], (3) feature regularization: Range Loss [59], (4) few-shot learning: FSLwF [13], (5) long-tailed ”
Few-Shot Learning Through an Information Retrieval Lens (2017, methodology): “These methods often employ hard-negative mining strategies for selecting informative constraints for training [10, 13].”
Decoupling Representation and Classifier for Long-Tailed Recognition (2019, methodology): “(Lin et al., 2017) 30.5 - - Range Loss† (Zhang et al., 2017) 30.7 - - Lifted Loss† (Oh Song et al., 2016) 30.8 - - OLTR† (Liu et al., 2019) 35 Liu et al., 2019), we found that using bigger backbone architectures increases performance signiﬁcantly on this dataset.”
In Defense of the Triplet Loss for Person Re-Identification (2017, background): “Furthermore we also show that a margin-less formulation performs slightly better, while removing one hyperparameter.”
Sampling Matters in Deep Embedding Learning (2017, methodology): “Our margin based loss and distance weighted sampling achieve state-of-the-art image retrieval and clustering performance on the Stanford Online Products, CARS196, and the CUB200-2011 datasets.”
Object-Centric Representation Learning from Unlabeled Videos (2016, background): “The idea of using triplets for learning embeddings has renewed interest lately [27,28,29] and can be traced to [30], which uses triplets to learn an image-to-image distance function that satisﬁes the property that the distance between images from the same category should be less than the distance…”
No Fuss Distance Metric Learning Using Proxies (2017, methodology): “Based on the experimental protocol detailed in [15, 14] we evaluate retrieval at k and clustering quality on data from unseen classes on 3 datasets: CUB200-2011 [17], Cars196 [6], and Stanford Online Products [8].”
Learning Metrics From Teachers: Compact Networks for Image Embedding (2019, methodology): “We follow the evaluation protocol proposed in [32].”
End-to-End Learning of Deep Visual Representations for Image Retrieval (2016, background): “Siamese networks have performedwell formetric learning (Song et al. 2016), dimensionality reduction (Hadsell et al.”