LQF: Linear Quadratic Fine-Tuning

By Alessandro Achille, Aditya Golatkar, Avinash Ravichandran, Marzia Polito, Stefano Soatto. 2020

33 citations recorded by the source corpus.

Classifiers that are linear in their parameters, and trained by optimizing a convex loss function, have predictable behavior with respect to changes in the training data, initial conditions, and optimization. Such desirable properties are absent in deep neural networks (DNNs), typically trained by non-linear fine-tuning of a pre-trained model. Previous attempts to linearize DNNs have led to interesting theoretical insights, but have not impacted the practice due to the substantial performance gap compared to standard non-linear optimization. We present the first method for linearizing a pre-trained model that achieves comparable performance to non-linear fine-tuning on most of real-world image classification tasks tested, thus enjoying the interpretability of linear models without incurring punishing losses in performance. LQF consists of simple modifications to the architecture, loss function and optimization typically used for classification: Leaky-ReLU instead of ReLU, mean squared loss instead of cross-entropy, and pre-conditioning using Kronecker factorization. None of these changes in isolation is sufficient to approach the performance of non-linear fine-tuning. When used in combination, they allow us to reach comparable performance, and even superior in the low-data regime, while enjoying the simplicity, robustness and interpretability of linear-quadratic optimization.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

DLCFT: Deep Linear Continual Fine-Tuning for General Incremental Learning (2022, methodology): “We apply first-order Taylor approximation of a pre-trained neural network [1, 31] which decomposes the feature extractor into a frozen non-linear network and a trainable linear network.”
Mixed-Privacy Forgetting in Deep Networks (2020, methodology): “Following the suggetsion of [2], we replace the ReLUs with Leaky ReLUs, since it boosts the accuracy of linearized models.”
TRAK: Attributing Model Behavior at Scale (2023, unknown): “While these works focus on neural networks in the their large or infinite-width limit, a line of recent works (Mu et al., 2020; Achille et al., 2021; Long, 2021; Atanasov et al., 2022; Wei et al., 2022; Malladi et al., 2022; Atanasov et al., 2023; Ma et al., 2022) studies instead the finite-width…”