Knowledge Flow: Improve Upon Your Teachers

By Iou-Jen Liu, Jian Peng, Alexander Gerhard Schwing. 2019

66 citations recorded by the source corpus.

A zoo of deep nets is available these days for almost any given task, and it is increasingly unclear which net to start with when addressing a new task, or which net to use as an initialization for fine-tuning a new model. To address this issue, in this paper, we develop knowledge flow which moves 'knowledge' from multiple deep nets, referred to as teachers, to a new deep net model, called the student. The structure of the teachers and the student can differ arbitrarily and they can be trained on entirely different tasks with different output spaces too. Upon training with knowledge flow the student is independent of the teachers. We demonstrate our approach on a variety of supervised and reinforcement learning tasks, outperforming fine-tuning and other 'knowledge exchange' methods.

Subfields: Artificial Intelligence, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Meta & Continual Learning

Citations in the corpus

Ensemble Distillation for Robust Model Fusion in Federated Learning (2020, methodology): “In contrast, ensemble learning methods (You et al., 2017; Furlanello et al., 2018; Anil et al., 2018; Dvornik et al., 2019; Park & Kwak, 2019; Liu et al., 2019; Wu et al., 2019) allow to combine multiple heterogeneous weak classifiers by averaging the predictions of the individual models instead.”
Knowledge Distillation: A Survey (2020, methodology): “Furthermore, the parameters of the teacher model (or the connections between layers) also contain another knowledge (Liu et al. 2019c).”
Distilling Knowledge From Graph Convolutional Networks (2020, background): “Knowledge amalgamation [33, 22, 32, 44] aims to learn a student network from multiple teachers from different domains.”
Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, methodology): “[131] focus on modeling the distribution of the parameters as prior knowledge, which is modeled by aggregating distribution (logits) space from the teacher network.”
Complementary Relation Contrastive Distillation (2021, background): “Knowledge distillation aims to transfer the knowledge from one deep learning model (the teacher) to another (the student), such as distilling a large network into a smaller one [19, 49, 2, 48, 12] or ensembling a collection of models into a single model [29, 37, 27, 45].”
Zoo-Tuning: Adaptive Transfer from a Zoo of Models (2021, methodology): “Some works leverage prior knowledge by inserting features from pretrained models (Rusu et al., 2016; Liu et al., 2019).”
Customizing Student Networks From Heterogeneous Teachers via Adaptive Knowledge Amalgamation (2019, background): “Knowledge flow [14] transfers knowledge from multiple teacher models to student one with strategy that student learns to predict with the help of teachers, but gradually reduce the dependency on teachers, finally predict independently.”