Fix your classifier: the marginal value of training the last weight layer

By Elad Hoffer, Itay Hubara, Daniel Soudry. 2018

107 citations recorded by the source corpus.

Neural networks are commonly used as models for classification for a wide variety of tasks. Typically, a learned affine transformation is placed at the end of such models, yielding a per-class value used for classification. This classifier can have a vast number of parameters, which grows linearly with the number of possible classes, thus requiring increasingly more resources. In this work we argue that this classifier can be fixed, up to a global scale constant, with little or no loss of accuracy for most tasks, allowing memory and computational benefits. Moreover, we show that by initializing the classifier with a Hadamard matrix we can speed up inference as well. We discuss the implications for current understanding of neural network models.

Subfields: Artificial Intelligence, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

Citations in the corpus

A Constructive Prediction of the Generalization Error Across Scales (2019, methodology): “For all other datasets we use WRN-44-16 (Wu et al., 2016) of varying widths, modified from the implementation of Hoffer et al. (2018).”
FRAGE: Frequency-Agnostic Word Representation (2018, background): “As the inputs of the neural network, word embeddings carry all the information of words that will be further processed by the network, and the quality of embeddings is critical and highly impacts the final performance of the learning task [16].”
Constrained Few-shot Class-incremental Learning (2022, methodology): “In fact, fixed quasi-orthogonal vectors have been successfully used as class vectors achieving improved performance in the supervised classification tasks [16, 17].”
Gradient Descent Finds Global Minima of Deep Neural Networks (2018, background): “Hoffer et al. [2018] show that fixing the last layer results in little to no loss in test accuracy.”
An Exponential Learning Rate Schedule for Deep Learning (2019, background): “However, Hoffer et al. (2018b) shows that ﬁxing the output layer randomly doesn’t harm the performance of the network.”
Asymmetric metric learning for knowledge transfer (2020, background): “BCT [60] ﬁxes the classiﬁer (last layer) of the student to that of the teacher, similarly to [24].”
Improving DNN Robustness to Adversarial Attacks using Jacobian Regularization (2018, methodology): “To provide a bound for this layer, we relied on the work in [27], which shows that fixating the weight matrix of the last fully connected layer in a deep neural network causes little to no loss of accuracy while allowing memory and computational benefits.”
Hyperspherical Prototype Networks (2019, unknown): “[15], who show the potential of fixed output spaces.”
Dense Classification and Implanting for Few-Shot Learning (2019, background): “Temperature scaling is frequently encountered in various formats in several works to enable soft-labeling [12] or to improve cosine similarity in the final layer [45, 30, 6, 31, 14].”
Dissecting Supervised Constrastive Learning (2021, background): “Overall, (Hoffer et al., 2018; Liu et al., 2018; Mettes et al., 2019) all control, in one way or the other, the geometric arrangement of classifier weights and thereby, implicitly, the arrangement of the representations.”
Regular Polytope Networks (2021, methodology): “This experiment also provides a novel and more systematic empirical evidence of the general applicability and correctness of fixed classifiers with respect to [1] where only one permutation was tested.”
Do We Need Fully Connected Output Layers in Convolutional Networks? (2020, methodology): “A few existing works have studied how to reduce the number of parameters in a CNN’s classifier for many-class datasets by using fixed output matrices [9, 26].”
No One Representation to Rule Them All: Overlapping Features of Training Methods (2021, background): “, 2019), and even features that fit a random, pre-determined classifier (Hoffer et al., 2018).”