Merging Models with Fisher-Weighted Averaging

By Michael Matena, Colin Raffel. 2021

671 citations recorded by the source corpus.

Averaging the parameters of models that have the same architecture and initialization can provide a means of combining their respective capabilities. In this paper, we take the perspective that this"merging"operation can be seen as choosing parameters that approximately maximize the joint likelihood of the posteriors of the models' parameters. Computing a simple average of the models' parameters therefore corresponds to making an isotropic Gaussian approximation to their posteriors. We develop an alternative merging procedure based on the Laplace approximation where we approximate each model's posterior as a Gaussian distribution whose precision matrix corresponds to its Fisher information. We first show that our"Fisher merging"technique provides a performance boost in settings where simple parameter averaging is currently used -- specifically, robust fine-tuning and model ensembling. Then, we compare merging to standard gradient-based transfer learning and demonstrate that merging enables a fundamentally different method for transferring capabilities across models. Specifically, we show that Fisher merging is competitive with gradient-based transfer learning approaches (while being significantly cheaper) in intermediate-task training and domain-adaptive pre-training. We also show that our merging procedure makes it possible to combine models in previously unexplored ways. We release our code to facilitate future research into methods for merging models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Optimization & Training

Citations in the corpus

Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time (2022, methodology): “Beyond ImageNet and associated distribution shifts, our results show that model soups are applicable when ﬁne-tuning on tasks from the WILDS (Koh et al., 2021) benchmark, and when ﬁne-tuning transformer models (Vaswani et al., 2017; Devlin et al., 2019a; Raffel et al., 2020b) for text classiﬁcation.”
VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance (2022, background): “The bulk of the existing literature on this topic focuses on editing or updating existing (typically unimodal) models in both computer vision [3] and natural language processing [8, 31, 28].”
Robust fine-tuning of zero-shot models (2021, methodology): “Finally, Matena and Raffel [66] introduce a method of combining models in weight-space that goes beyond linear interpolation with a single mixing-coefficient as employed in WiSE-FT.”
TIES-Merging: Resolving Interference When Merging Models (2023, methodology): “Fisher Merging [45] goes beyond simple averaging to identify the importance of individual parameters using Fisher Information Matrix [18, 3, 34] and uses it to weigh the parameters in each model when merging.”
Dataless Knowledge Fusion by Merging Weights of Language Models (2022, neutral): “Model merging is defined as combining multiple models into a single one in parameter space without access to data (Matena & Raffel, 2021).”
Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models (2022, positive): “As an alternative to ensembling, we can also use parameter averaging (Izmailov et al., 2018; Wortsman et al., 2022a; Matena and Raffel, 2021) to collapse the ELMFOREST into a single LM.”
Editing Models with Task Arithmetic (2022, positive): “Moreover, adding a task vector from a different task can improve performance on a target task using text models (T5, Raffel et al. [79]).”
Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models (2024, background): “Prior works have demonstrated that weight merging can enhance performance across various domains, including language modeling (Li et al., 2022; Matena and Raffel, 2022; Ilharco et al., 2022; Don-Yehiya et al., 2022; Gururangan et al., 2023; Yadav et al., 2024; Sukhbaatar et al., 2024),…”
Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora (2021, background): “, 2019a) or directly merging models in the parameter space (Matena and Raffel, 2021).”
Git Re-Basin: Merging Models modulo Permutation Symmetries (2022, background): “We therefore anticipate its potential application in ﬁelds such as ﬁnetuning (Devlin et al., 2019; Wortsman et al., 2022b;a), federated learning (McMa-han et al., 2017; Kone ˇ cn ´ y et al., 2016a;b), and model patching (Matena & Raffel, 2021; Sung et al., 2021; Raffel, 2021).”
Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch (2023, positive): “Following Matena & Raffel (2022); Jin et al. (2023); Yadav et al. (2023), we focus on merging fine-tuned models that are optimized from the same pre-trained backbone.”
LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition (2023, neutral): “Prior works (Matena & Raffel, 2021; Jin et al., 2023; Yadav et al., 2023; Wu et al., 2023a) .”
Statistical inference on black-box generative models in the data kernel perspective space (2024, unknown): “Other meth-ods such as retrieval-augmented generation (Lewis et al., 2020), model merging (Matena and Raffel, 2022), constrained decoding (Hokamp and Liu, 2017), etc. (Dettmers et al., 2024; Edge et al., 2024), have similarly contributed to the rapid development of a large population of diverse and…”
Query-efficient model evaluation using cached responses (2026, unknown): “…ever to produce new model behavior – via parameter efficient fine-tuning (Han et al., 2024), instruction tuning (Zhang et al., 2025), model merging (Matena & Raffel, 2022), distillation (Gou et al., 2021), prompt engineering (Sahoo et al., 2025), etc. – making it infeasible to evaluate each new…”