Learning Neural Network Subspaces

By Mitchell Wortsman, Maxwell Horton, Carlos Guestrin, Ali Farhadi, Mohammad Rastegari. 2021

104 citations recorded by the source corpus.

Recent observations have advanced our understanding of the neural network optimization landscape, revealing the existence of (1) paths of high accuracy containing diverse solutions and (2) wider minima offering improved performance. Previous methods observing diverse paths require multiple training runs. In contrast we aim to leverage both property (1) and (2) with a single method and in a single training run. With a similar computational cost as training one model, we learn lines, curves, and simplexes of high-accuracy neural networks. These neural network subspaces contain diverse solutions that can be ensembled, approaching the ensemble performance of independently trained networks without the training cost. Moreover, using the subspace midpoint boosts accuracy, calibration, and robustness to label noise, outperforming Stochastic Weight Averaging.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

CoSCL: Cooperation of Small Continual Learners is Stronger than a Big One (2022, positive): “They further exploited recent advances of mode connectivity [30] and neural network subspace [48] to save computational cost, but had to use old training samples [13].”
Towards Better Plasticity-Stability Trade-off in Incremental Learning: A Simple Linear Connector (2021, positive): “Observation 1 (Connectivity) [10,14] There exists a continuous path between minima of neural network architectures, where each point along this path has a low loss.”
Robust fine-tuning of zero-shot models (2021, background): “…ﬁne-tuned models, i.e., there exists an α for which We are not the ﬁrst to observe that when interpolating between models with linear mode connectivity, the accuracy of models along the path may exceed that of either endpoint (Izmailov et al., 2018; Neyshabur et al., 2020; Wortsman et al., 2021).”
Editing Models with Task Arithmetic (2022, unknown): “The geometry of neural network loss surfaces has attracted the interest of several authors in recent years [54; 28; 21; 48; 25; 13; 98; 7; 23; 55; 60].”
Building a Subspace of Policies for Scalable Continual Learning (2022, background): “The additional computational cost compared to a single network is negligible during both training and inference as it is mentioned in Wortsman et al. (2021).”
The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks (2021, background): “Motivated to create ensembles, Wortsman et al. (2021) start with two (or more) random initializations and learn a subspace (a line or simplex) connecting them.”
Efficient Continual Learning Ensembles in Neural Network Subspaces (2022, methodology): “For the subspace method training, we proceed similarly as in Wortsman et al. (2021).”