Asymmetric metric learning for knowledge transfer

By Mateusz Budnik, Yannis Avrithis. 2020

42 citations recorded by the source corpus.

Knowledge transfer from large teacher models to smaller student models has recently been studied for metric learning, focusing on fine-grained classification. In this work, focusing on instance-level image retrieval, we study an asymmetric testing task, where the database is represented by the teacher and queries by the student. Inspired by this task, we introduce asymmetric metric learning, a novel paradigm of using asymmetric representations at training. This acts as a simple combination of knowledge transfer with the original metric learning task.We systematically evaluate different teacher and student models, metric learning and knowledge transfer loss functions on the new asymmetric testing as well as the standard symmetric testing task, where database and queries are represented by the same model. We find that plain regression is surprisingly effective compared to more complex knowledge transfer mechanisms, working best in asymmetric testing. Interestingly, our asymmetric metric learning approach works best in symmetric testing, allowing the student to even out-perform the teacher.Our implementation is publicly available,1 including trained student models for all loss functions and all pairs of teacher/student models.2

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Han-Jia Ye
2. De-chuan Zhan
3. Da-Wei Zhou
4. Fu-Yun Wang
5. Shiliang Pu
6. Liang Ma
7. Qiwen Wang
8. Ziwei Liu

In Computer Vision and Pattern Recognition

1. Han-Jia Ye
2. De-chuan Zhan
3. Da-Wei Zhou
4. Fu-Yun Wang
5. Liang Ma
6. Shiliang Pu
7. Qiwen Wang
8. Ziwei Liu

Citations in the corpus

• Forward Compatible Few-Shot Class-Incremental Learning (2022, background): “Recent work focuses on improving model’s backward compatibility [7, 28, 35], while we are the first to address the model’s forward compatibility in FSCIL.”
• Forward Compatible Training for Large-Scale Embedding Retrieval Systems (2021, unknown): “Recently, [4] also observed superior performance using L2 loss between features for knowledge distillation.”