Learning Metrics From Teachers: Compact Networks for Image Embedding

By Lu Yu, V. O. Yazici, Xialei Liu, Joost van de Weijer, Yongmei Cheng, Arnau Ramisa. 2019

123 citations recorded by the source corpus.

Metric learning networks are used to compute image embeddings, which are widely used in many applications such as image retrieval and face recognition. In this paper, we propose to use network distillation to efficiently compute image embeddings with small networks. Network distillation has been successfully applied to improve image classification, but has hardly been explored for metric learning. To do so, we propose two new loss functions that model the communication of a deep teacher network to a small student network. We evaluate our system in several datasets, including CUB-200-2011, Cars-196, Stanford Online Products and show that embeddings computed using small student networks perform significantly better than those computed using standard networks of similar size. Results on a very compact network (MobileNet-0.25), which can be used on mobile devices, show that the proposed method can greatly improve Recall@1 results from 27.5\% to 44.6\%. Furthermore, we investigate various aspects of distillation for embeddings, including hint and attention layers, semi-supervised learning and cross quality distillation. (Code is available at https://github.com/yulu0724/EmbeddingDistillation).

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Efficiency & Compression

Citations in the corpus

Knowledge Distillation: A Survey (2020, background): “ionships between diﬀerent activations, neurons or pairs of samples contain rich information learned by the teacher model (Yim et al., 2017; Lee and Song, 2019; Liu et al., 2019f; Tung and Mori, 2019; Yu et al., 2019). Furthermore, the parameters of the teacher model (or the connections between layer”
Generative Feature Replay For Class-Incremental Learning (2020, methodology): “Note that feature distillation has also been used in other applications [32, 47, 55].”
Semantic Drift Compensation for Class-Incremental Learning (2020, methodology): “Instead, on embedding networks, we constrain the parameters drift by minimizing the distance between the output embeddings of image x i during training the current task ( z t i ) with respect to its embedding in the previous task ( z t − 1 i ), similar as [48].”
Knowledge Distillation for Incremental Learning in Semantic Segmentation (2019, methodology): “Another strategy is to use network distillation to e ﬃ ciently compute image embeddings with small networks for metric learning [41].”
Probabilistic Knowledge Transfer for Lightweight Deep Representation Learning (2020, methodology): “Finally, in [20], an embedding-based approach was proposed for transferring the knowledge, while in [24], a multidimensional-scaling-based method was used to learn a student that maintains the same distances as the teacher between the pairs of samples.”
Creating Something From Nothing: Unsupervised Knowledge Distillation for Cross-Modal Hashing (2020, methodology): “Recently, many efforts were made to study what the dark knowledge is [41, 40], and/or how to efficiently take advantage of such knowledge [11, 42, 35, 2].”
Universal Representation Learning from Multiple Domains for Few-shot Classification (2021, methodology): “Here we go beyond the few-shot classification experiments and evaluate the generalization ability of our representations that are learned in the multi-domain network in a retrieval task, inspired from metric learning literature [37, 55].”
Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, background): “[279] since both aim to measure the distance between the teacher’s and the student’s embeddings.”
Heterogeneous Knowledge Distillation Using Information Flow Modeling (2020, methodology): “The proposed method was eval-Table [26] and SUN Attribute [18] datasets) and compared to four competitive KD methods: neural network distillation [9], hint-based transfer [19], probabilistic knowledge transfer (PKT) [16] and metric knowledge transfer (abbreviated as MKT) [28].”
Knowledge Transfer Graph for Deep Collaborative Learning (2019, background): “[22, 31, 19] transfer mutual relations of data samples in a mini-batch.”
Asymmetric metric learning for knowledge transfer (2020, background): “It is the same as the absolute version of metric learning knowledge distillation (ML+KD) [82] and as contrastive loss (4) on asymmetric similarity (3) (using only the anchor as a positive for itself ).”
Collaborative Teacher-Student Learning via Multiple Knowledge Transfer (2021, background): “Unlike both the response-based and the feature-based knowledge from individual instances, the relation-based knowledge from instance relations is modelled for improving student learning [37, 38, 15, 16, 17].”