DarkRank: Accelerating Deep Metric Learning via Cross Sample Similarities Transfer

By Yuntao Chen, Naiyan Wang, Zhaoxiang Zhang. 2017

236 citations recorded by the source corpus.

We have witnessed rapid evolution of deep neural network architecture design in the past years. These latest progresses greatly facilitate the developments in various areas such as computer vision and natural language processing. However, along with the extraordinary performance, these state-of-the-art models also bring in expensive computational cost. Directly deploying these models into applications with real-time requirement is still infeasible. Recently, Hinton et al. have shown that the dark knowledge within a powerful teacher model can significantly help the training of a smaller and faster student network. These knowledge are vastly beneficial to improve the generalization ability of the student model. Inspired by their work, we introduce a new type of knowledge---cross sample similarities for model compression and acceleration. This knowledge can be naturally derived from deep metric learning model. To transfer them, we bring the "learning to rank" technique into deep metric learning formulation. We test our proposed DarkRank method on various metric learning tasks including pedestrian re-identification, image retrieval and image clustering. The results are quite encouraging. Our method can improve over the baseline method by a large margin. Moreover, it is fully compatible with other existing methods. When combined, the performance can be further boosted.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Knowledge Distillation: A Survey (2020, methodology): “. A teacher supervised the student as user-item prediction network (generator). The student learning was adjusted by adversarial adaption between teacher and student networks. Unlike distillation in (Chen et al., 2018b; Tang and Wang, 2018), Pan et al. (2019) designed a enhanced collaborative denois”
Relational Knowledge Distillation (2019, methodology): “Furthermore, our experiments on metric learning show that the proposed method outperforms [7] with a significant margin.”
Learning Metrics From Teachers: Compact Networks for Image Embedding (2019, methodology): “[4] bring the ’learning to rank’ technique into deep metric learning for knowledge transfer.”
Towards Oracle Knowledge Distillation with Neural Architecture Search (2019, methodology): “KD is originally proposed to learn compact and fast models and has widely been applied to many practical applications including object detection (Chen et al. 2017; Li, Jin, and Yan 2017), face recognition (Luo et al. 2016) and image retrieval (Chen, Wang, and Zhang 2018).”
Prime-Aware Adaptive Distillation (2020, methodology): “ing, thus, we choose the embedding feature as the distillation target 10 Y. et al. layer for all methods. There are few works designed specically for metric learning. We choose typical L 2, Darkrank [5] and RKD [24] as the compared methods. Two losses are proposed in Darkrank [5], i.e., HardRank and”
AlignedReID: Surpassing Human-Level Performance in Person Re-Identification (2017, background): “DarkRank [4] introduces a new type of knowledge-cross sample similarity for model compression and acceleration, achieving state-of-the-art performance.”
A Pose-Sensitive Embedding for Person Re-identification with Expanded Cross Neighborhood Re-ranking (2017, background): “We studied the impact of changing these on Market and Duke datasets and found that it is subtle in the range for t ∈ [2 , 4] and q ∈ [4 , 10] , the performance drops between ∼ 0 .”
GAN Compression: Efficient Architectures for Interactive Conditional GANs (2020, methodology): “A widely-used method for model compression is knowledge distillation, which matches the distribution of the output layer’s logits [23, 43, 9, 63, 33, 46, 10].”
Exclusivity-Consistency Regularized Knowledge Distillation for Face Recognition (2020, background): “Some studies [23, 2, 40, 4, 1, 54] extended knowledge distillation to other applications.”
A Survey of Model Compression and Acceleration for Deep Neural Networks (2017, unknown): “tly reducing and transferring parameters from the teacher models, passing selectivity knowledge of neurons could be helpful. One can derive a way to select essential neurons related to the task [77], [78]. The intuition is that if a neuron is activated in certain regions or samples, that implies the”
Knowledge Squeezed Adversarial Network Compression (2019, unknown): “Various derivatives have been implemented in different tasks: object detection [31], semantic segmentation [32], and metric learning [33], etc.”
Knowledge Distillation with Adversarial Samples Supporting Decision Boundary (2018, background): “Some studies [4, 19, 5] extended knowledge distillation to computer vision applications.”
Highlight Every Step: Knowledge Distillation via Collaborative Teaching (2019, methodology): “Moreover, KD also has been applied to other applications, such as object detection [68], pedestrian reidentification [69],”
Interactive Knowledge Distillation (2020, background): “dded into resource-constrained devices, e.g., mobiles and UAVs. To reduce the resource consumption while maintaining good performance, researchers leverage knowledge distillation techniques [6], [7], [8], [9], [10], [11] to transfer informative knowledge from a cumbersome but well-trained teacher ne”
Creating Something From Nothing: Unsupervised Knowledge Distillation for Cross-Modal Hashing (2020, methodology): “In particular, DarkRank [5] distilled knowledge for deep metric learning by matching two probability distributions over ranking, while our approach utilized knowledge by selecting relevant instances.”
Training Student Networks for Acceleration with Conditional Adversarial Networks (2018, methodology): “Variants of knowledge distillation has also been applied to tasks in other domains [5, 23, 30, 33] A recent preprint [17] presented promising preliminary results on CIFAR-10 by learning a small ResNet from a large ResNet.”
Training Shallow and Thin Networks for Acceleration via Knowledge Distillation with Conditional Adversarial Networks (2017, methodology): “Variants of knowledge distillation has also been applied to many different tasks, such as semantic segmentation [31], pedestrian detection [35], face recognition [24], metric learning[6], reinforcement learning [38] and for regularization[34].”