Transferring Inductive Biases through Knowledge Distillation

By Samira Abnar, Mostafa Dehghani, Willem H. Zuidema. 2020

72 citations recorded by the source corpus.

Having the right inductive biases can be crucial in many tasks or scenarios where data or computing resources are a limiting factor, or where training data is not perfectly representative of the conditions at test time. However, defining, designing and efficiently adapting inductive biases is not necessarily straightforward. In this paper, we explore the power of knowledge distillation for transferring the effect of inductive biases from one model to another. We consider families of models with different inductive biases, LSTMs vs. Transformers and CNNs vs. MLPs, in the context of tasks and scenarios where having the right inductive biases is critical. We study how the effect of inductive biases is transferred through knowledge distillation, in terms of not only performance but also different aspects of converged solutions.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

Citations in the corpus

Training data-efficient image transformers & distillation through attention (2020, methodology): “KD can transfer inductive biases [1] in a soft way in a student model using a teacher model where they would be incorporated in a hard way.”
ConViT: improving vision transformers with soft convolutional inductive biases (2021, neutral): “In ﬁgure 10 of the appendix, we also show that the nonlocality metric is lower when training with distillation from a convolutional network as in Touvron et al (2020), suggesting that the locality of the teacher is partly transferred to the student (Abnar et al 2020).”
Self-Distillation Amplifies Regularization in Hilbert Space (2020, background): “[1] empirically studies how inductive biases are transferred through distillation.”
Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling? (2022, background): “It is also good to consider if different inductive biases will result in different extents of emergent capabilities (Wei et al., 2022a; Abnar et al., 2020).”
The Benchmark Lottery (2021, methodology): “As an example, the subject-verb agreement task [Linzen et al., 2016] became more popular, when it started being used for showcasing the ability of recurrent neural networks over Transformers on modeling hierarchical structure [Tran et al., 2018, Dehghani et al., 2019, Abnar et al., 2020].”
Efficient Transformers in Reinforcement Learning using Actor-Learner Distillation (2021, background): “Similar to this work’s desired goal to leverage the rapid memorization capabilities of transformers while training an LSTM, other work showed that distillation between different architecture classes could enable the transfer of inductive biases (Kuncoro et al., 2019; Abnar et al., 2020).”
Forward Compatible Training for Large-Scale Embedding Retrieval Systems (2021, background): “Further, it can potentially carry unwanted biases from the old to the new model [1].”
A Survey of Visual Transformers (2021, background): “These observations are explained in [172]: the teacher CNN transfers its inductive bias in a soft way to the student Transformer through knowledge distillation.”