The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks

By Rahim Entezari, Hanie Sedghi, Olga Saukh, Behnam Neyshabur. International Conference on Learning Representations, 2021

334 citations recorded by the source corpus.

In this paper, we conjecture that if the permutation invariance of neural networks is taken into account, SGD solutions will likely have no barrier in the linear interpolation between them. Although it is a bold conjecture, we show how extensive empirical attempts fall short of refuting it. We further provide a preliminary theoretical result to support our conjecture. Our conjecture has implications for lottery ticket hypothesis, distributed training, and ensemble methods.

Subfields: Artificial Intelligence, Management Science and Operations Research

View Paper: https://www.semanticscholar.org/paper/4dc48bd4e1c0e5986b36eca8339bd45e944d8a82

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

Citations in the corpus

Editing Models with Task Arithmetic (2022, positive): “The geometry of neural network loss surfaces has attracted the interest of several authors in recent years [54; 28; 21; 48; 25; 13; 98; 7; 23; 55; 60].”
Git Re-Basin: Merging Models modulo Permutation Symmetries (2022, methodology): “Following Frankle et al. (2020); Entezari et al. (2021); Draxler et al. (2018); Garipov et al. (2018) and others, we measure approximations to convexity via “barriers.””
Transformers learn in-context by gradient descent (2022, methodology): “We take inspiration on recent work (Benzing et al., 2022; Entezari et al., 2021) that showed approximate equivalence of models found by SGD after permuting weights within the trained neural networks.”
Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models (2022, positive): “Seed training Our discovery of the importance of the seed training as a critical warm-up phase for BTM is in line with ﬁndings that parameter averaging only works when models share part of their optimization trajectory (Frankle et al., 2020; Entezari et al., 2022”
TIES-Merging: Resolving Interference When Merging Models (2023, background): “Entezari et al. [16] hypothesized that if we account for the permutation symmetry of neural networks, then all neural networks of a given architecture trained on the same dataset are linear mode connected.”