Towards Understanding the Transferability of Deep Representations

By Hong Liu, Mingsheng Long, Jianmin Wang, Michael I. Jordan. 2019

28 citations recorded by the source corpus.

Deep neural networks trained on a wide range of datasets demonstrate impressive transferability. Deep features appear general in that they are applicable to many datasets and tasks. Such property is in prevalent use in real-world applications. A neural network pretrained on large datasets, such as ImageNet, can significantly boost generalization and accelerate training if fine-tuned to a smaller target dataset. Despite its pervasiveness, few effort has been devoted to uncovering the reason of transferability in deep feature representations. This paper tries to understand transferability from the perspectives of improved generalization, optimization and the feasibility of transferability. We demonstrate that 1) Transferred models tend to find flatter minima, since their weight matrices stay close to the original flat region of pretrained parameters when transferred to a similar target dataset; 2) Transferred representations make the loss landscape more favorable with improved Lipschitzness, which accelerates and stabilizes training substantially. The improvement largely attributes to the fact that the principal component of gradient is suppressed in the pretrained parameters, thus stabilizing the magnitude of gradient in back-propagation. 3) The feasibility of transferability is related to the similarity of both input and label. And a surprising discovery is that the feasibility is also impacted by the training stages in that the transferability first increases during training, and then declines. We further provide a theoretical analysis to verify our observations.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Generalisation Guarantees for Continual Learning with Orthogonal Gradient Descent (2020, methodology): “Remark 1 Corollary 2 can be applied to get a similar result to Theorem 3 by Liu et al. (2019).”
TranSlider: Transfer Ensemble Learning from Exploitation to Exploration (2020, background): “(3), we assume a prior distribution on the parameters ?̂?t by borrowing recent observations in transfer learning that the optimal solution ?̂?t usually lies around ωs after fine-tuning to the target [21, 23].”
Transfer Learning with Style Transfer between the Photorealistic and Artistic Domain (2021, background): “However, similarity of target tasks also matters for the transferability of pretrained networks [55] that may lead to negative transfer [16].”
RIFLE: Backpropagation in Depth for Deep Transfer Learning through Re-Initializing the Fully-connected LayEr (2020, methodology): “, 2019), since the learning dynamics would converge to a local minima of loss that is close to its starting point (Neyshabur et al., 2015; Soudry et al., 2018; Liu et al., 2019).”