DEMix Layers: Disentangling Domains for Modular Language Modeling

By Suchin Gururangan, Michael Lewis, Ari Holtzman, Noah A. Smith, Luke S. Zettlemoyer. 2021

183 citations recorded by the source corpus.

We introduce a new domain expert mixture (DEMix) layer that enables conditioning a language model (LM) on the domain of the input text. A DEMix layer includes a collection of expert feedforward networks, each specialized to a domain, that makes the LM modular: experts can be mixed, added, or removed after initial training. Extensive experiments with autoregressive transformer LMs (up to 1.3B parameters) show that DEMix layers reduce test-time perplexity (especially for out-of-domain data), increase training efficiency, and enable rapid adaptation. Mixing experts during inference, using a parameter-free weighted ensemble, enables better generalization to heterogeneous or unseen domains. We also show it is possible to add experts to adapt to new domains without forgetting older ones, and remove experts to restrict access to unwanted domains. Overall, these results demonstrate benefits of domain modularity in language models.

Subfields: Artificial Intelligence, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Natural Language Processing

Citations in the corpus

A Review of Sparse Expert Models in Deep Learning (2022, background): “Additionally, the appropriate granularity of sparsity still must be determined: most works have focused on experts replacing components, such as feed-forward network layers, but beneﬁts of more fully modular, independent experts were discovered (Gururangan et al., 2021; Li et al., 2022).”
Continual Training of Language Models for Few-Shot Learning (2022, background): “(6) DEMIX (Gururangan et al., 2021) is a recent model to adapt a pre-trained LM with new domains.”
ELLE: Efficient Lifelong Pre-training for Emerging Data (2022, background): “Some target at domain adaptation through continual pre-training (Gururangan et al., 2020), parameter-efﬁcient adapters (He et al., 2021) and sparse expert models (Gururangan et al., 2021).”
Continual Pre-training of Language Models (2023, background): “Those baselines (BCL, CLASSIC and DEMIX) perform knowledge transfer achieve better results but still poorer than DAS.”
ST-MoE: Designing Stable and Transferable Sparse Expert Models (2022, background): “Finally, Gururangan et al. (2021) recommends even greater modularity of language models and conditionally activates experts based on the domain/task label or by an inferred label.”
A Survey on Mixture of Experts in Large Language Models (2024, unknown): “This preference is due to the fact that tensor parallelism typically incurs a relatively higher communication overhead.”
Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models (2022, unknown): “This is identical to the DENSE model from Gururangan et al. (2022), in which data from each domain is balanced.”
Time Waits for No One! Analysis and Challenges of Temporal Misalignment (2021, background): “Recent work in language modeling has elevated the importance of domains by using a mixture of domains (Gururangan et al., 2021) or giving domains a hierarchical structure (Chronopoulou et al.”
SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore (2023, background): “Our work is closely related to recent studies on modular LMs, which have specialized parameters (or experts) trained on different domains (Gururangan et al., 2022; Li et al., 2022; Gururangan et al., 2023), languages (Pfeiffer et al.”
Lifting the Curse of Multilinguality by Pre-training Modular Transformers (2022, methodology): “All in070 all, we propose a multilingual architecture that can071 scale to a large number of languages without any072 loss in performance, and can be further extended073 to new languages after pre-training.1074 2 Background and Related Work075 We provide a background on modular and multi-076 lingual”
Continual Learning of Large Language Models: A Comprehensive Survey (2024, unknown): “Extensive studies have not only demonstrated the necessity of CPT for improved downstream performance [35, 71, 95, 96, 100, 184], but also shown that when distributional shifts are gradual [95, 278] or somewhat correlated [71], CPT can effectively help model generalize to unseen data.”
Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora (2021, background): “, 2021), including learning domain specific expert models (Gururangan et al., 2021), are also actively studied.”
Designing Effective Sparse Expert Models (2022, background): “Finally, Gururangan et al. (2021) recommends even greater modularity of language models and conditionally activates experts based on the domain/task label or by an inferred label.”