Ranking Distillation: Learning Compact Ranking Models With High Performance for Recommender System

By Jiaxi Tang, Ke Wang. 2018

231 citations recorded by the source corpus.

We propose a novel way to train ranking models, such as recommender systems, that are both effective and efficient. Knowledge distillation (KD) was shown to be successful in image recognition to achieve both effectiveness and efficiency. We propose a KD technique for learning to rank problems, called ranking distillation (RD). Specifically, we train a smaller student model to learn to rank documents/items from both the training data and the supervision of a larger teacher model. The student model achieves a similar ranking performance to that of the large teacher model, but its smaller model size makes the online inference more efficient. RD is flexible because it is orthogonal to the choices of ranking models for the teacher and student. We address the challenges of RD for ranking problems. The experiments on public data sets and state-of-the-art recommendation models showed that RD achieves its design purposes: the student model learnt with RD has less than an half size of the teacher model while achieving a ranking performance similar tothe teacher model and much better than the student model learnt without RD.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Knowledge Distillation: A Survey (2020, background): “Recently, knowledge distillation has been successfully applied in recommender systems for deep model compression and acceleration (Chen et al., 2018b; Tang and Wang, 2018; Pan et al., 2019).”
Deep Learning Based Recommender System (2017, background): “A promising area of research in this area involves knowledge distillation which have been explored in [144] for learning small/compact models for inference in recommender systems.”
Understanding and Improving Knowledge Distillation (2020, background): “Recently, we have witnessed a huge success of knowledge distillation, irrespective of the model architecture and application domain (Kim & Rush, 2016; Chen et al., 2017; Tang & Wang, 2018; Anil et al., 2018; He et al., 2019).”
Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation (2020, background): “In the IR setting, Tang and Wang [38] distill sequential recommendation models for recommender systems with one teacher model.”
Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling (2021, methodology): “Apart from specifically training dense retrieval models, knowledge distillation gains popularity, with general-purpose BERT-style models [17, 33] as well as a range of applications in IR: from sequential recommendation models [35], BERT-based retrieval chatbots [36], BERT-based Question Answering [1”