Gradients as Features for Deep Representation Learning

By Fangzhou Mu, Yingyu Liang, Yin Li. 2020

46 citations recorded by the source corpus.

We address the challenging problem of deep representation learning--the efficient adaption of a pre-trained deep network to different tasks. Specifically, we propose to explore gradient-based features. These features are gradients of the model parameters with respect to a task-specific loss given an input sample. Our key innovation is the design of a linear model that incorporates both gradient features and the activation of the network. We show that our model provides a local linear approximation to a underlying deep model, and discuss important theoretical insight. Moreover, we present an efficient algorithm for the training and inference of our model without computing the actual gradients. Our method is evaluated across a number of representation learning tasks on several datasets and using different network architectures. We demonstrate strong results in all settings. And our results are well-aligned with our theoretical insight.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

DLCFT: Deep Linear Continual Fine-Tuning for General Incremental Learning (2022, methodology): “We apply first-order Taylor approximation of a pre-trained neural network [1, 31] which decomposes the feature extractor into a frozen non-linear network and a trainable linear network.”
Mixed-Privacy Forgetting in Deep Networks (2020, methodology): “We observe that a straightforward application of [33] to a standard pre-trained ResNet50 tend to under-perform in our setting (fine-tuning on large scale vision tasks).”
LQF: Linear Quadratic Fine-Tuning (2020, methodology): “Work on the Neural Tangent Kernel (NTK) showed that linear models approximate the dynamics of randomly initialized deep networks [19, 28], at least when the number of filters goes to infinity, although [36] argue the analysis is valid for finite networks in the case of fine-tuning when the weights w”
TRAK: Attributing Model Behavior at Scale (2023, unknown): “While these works focus on neural networks in the their large or infinite-width limit, a line of recent works (Mu et al., 2020; Achille et al., 2021; Long, 2021; Atanasov et al., 2022; Wei et al., 2022; Malladi et al., 2022; Atanasov et al., 2023; Ma et al., 2022) studies instead the finite-width…”