Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch

By Le Yu, Yu Bowen, Haiyang Yu, Fei Huang, Yongbin Li. International Conference on Machine Learning, 2023

722 citations recorded by the source corpus.

In this paper, we unveil that Language Models (LMs) can acquire new capabilities by assimilating parameters from homologous models without retraining or GPUs. We first introduce DARE to set most delta parameters (i.e., the disparity between fine-tuned and pre-trained parameters) to zeros without affecting the abilities of Supervised Fine-Tuning (SFT) LMs, which randomly Drops delta parameters with a ratio $p$ And REscales the remaining ones by $1 / (1 - p)$ to approximate the original embeddings. Then, we use DARE as a versatile plug-in to sparsify delta parameters of multiple SFT homologous models for mitigating parameter interference and merge them into a single model by parameter fusing. We experiment with encoder- and decoder-based LMs, showing that: (1) SFT delta parameter value ranges are typically small (within 0.002) with extreme redundancy, and DARE can effortlessly eliminate 90% or even 99% of them; (2) DARE can merge multiple task-specific LMs into one LM with diverse capabilities. Notably, this phenomenon is more pronounced in large-scale LMs, where the merged LM reveals the potential to surpass the performance of any source LM, providing a new discovery. We also utilize DARE to create a merged LM that ranks first among models with 7 billion parameters on the Open LLM Leaderboard.

View Paper: https://www.semanticscholar.org/paper/c0230760f644f6b7538d93e4296a5e9aa7028e45

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Sean Welleck
Graham Neubig
Shayne Longpre
Seungone Kim
Juyoung Suk
Kyungjae Lee
Bill Yuchen Lin
Jamin Shin

Citations in the corpus

BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains (2024, methodology): “Among these methods, we can cite TIES (Ya-dav et al., 2023), DARE (Yu et al., 2024), and SLERP (Shoemake, 1985).”
Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models (2024, methodology): “For example, linear merging is as follows: In addition to linear merging, we test 5 additional variants, namely Task Arithmetic merging (Ilharco et al., 2022), TIES merging (Yadav et al., 2024), DARE-TIES and DARE-Linear merging (Yu et al., 2023), and SLERP merging (Goddard et al., 2024).”