Git Re-Basin: Merging Models modulo Permutation Symmetries

By Samuel K. Ainsworth, Jonathan Hayase, Siddhartha S. Srinivasa. 2022

518 citations recorded by the source corpus.

The success of deep learning is due in large part to our ability to solve certain massive non-convex optimization problems with relative ease. Though non-convex optimization is NP-hard, simple algorithms -- often variants of stochastic gradient descent -- exhibit surprising effectiveness in fitting large neural networks in practice. We argue that neural network loss landscapes often contain (nearly) a single basin after accounting for all possible permutation symmetries of hidden units a la Entezari et al. 2021. We introduce three algorithms to permute the units of one model to bring them into alignment with a reference model in order to merge the two models in weight space. This transformation produces a functionally equivalent set of weights that lie in an approximately convex basin near the reference model. Experimentally, we demonstrate the single basin phenomenon across a variety of model architectures and datasets, including the first (to our knowledge) demonstration of zero-barrier linear mode connectivity between independently trained ResNet models on CIFAR-10. Additionally, we identify intriguing phenomena relating model width and training time to mode connectivity. Finally, we discuss shortcomings of the linear mode connectivity hypothesis, including a counterexample to the single basin theory.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

On The Impact of Machine Learning Randomness on Group Fairness (2023, background): “Another possible explanation can be built on a recent line of work that shows there is only one functionally unique minima in loss landscape of neural networks, while all other minima simply contain permutation or scale symmetries of the same set of models [2].”
Editing Models with Task Arithmetic (2022, unknown): “Inspired by recent work on weight interpolation [27; 100; 63; 99; 39; 55; 2; 20], we obtain such vectors by taking the weights of a model fine-tuned on a task and subtracting the corresponding pre-trained weights (Figure 1a).”
Dataless Knowledge Fusion by Merging Weights of Language Models (2022, unknown): “Further, improved merging algorithms aim to match permutations between the weights of different models (Singh & Jaggi, 2020; Nguyen et al., 2021; Ainsworth et al., 2022; Wang et al., 2020a), which is a complementary line of effort to our work.”
The Platonic Representation Hypothesis (2024, unknown): “This holds even for models with different initial-izations, up to permutations over weight space (Ainsworth et al., 2022).”
BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains (2024, methodology): “Methods include averaging model weights or considering permutation invariance (Il-harco et al., 2022; Choshen et al., 2022; Singh and Jaggi, 2020; Ainsworth et al., 2023).”