Hash Layers For Large Sparse Models

By Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam, Jason Aaron Edward Weston. 2021

301 citations recorded by the source corpus.

We investigate the training of sparse layers that use different parameters for different inputs based on hashing in large Transformer models. Specifically, we modify the feedforward layer to hash to different sets of weights depending on the current token, over all tokens in the sequence. We show that this procedure either outperforms or is competitive with learning-to-route mixture-of-expert methods such as Switch Transformers and BASE Layers, while requiring no routing parameters or extra terms in the objective function such as a load balancing loss, and no sophisticated assignment algorithm. We study the performance of different hashing techniques, hash sizes and input features, and show that balanced and random hashes focused on the most local features work best, compared to either learning clusters or using longer-range context. We show our approach works well both on large language modeling and dialogue tasks, and on downstream fine-tuning tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Unified Scaling Laws for Routed Language Models (2022, methodology): “We analyze three different techniques for training Routing Networks, detailed in §3: Sinkhorn-BASE, a sparse mixture-of-experts (SMOE) approach modifying BASE [Lewis et al., 2021]; non-parametric HASH Layers [Roller et al., 2021]; and routing via Reinforcement Learning (RL-R).”
Efficient Large Scale Language Modeling with Mixtures of Experts (2021, methodology): “, 2021) have been successfully used for language modeling and machine translation (Lepikhin et al., 2021; Lewis et al., 2021; Roller et al., 2021), but are yet to be shown effective for fine-tuning (Fedus et al.”
A Review of Sparse Expert Models in Deep Learning (2022, background): “Most routing algorithms are dynamic, but a notable example of a static routing algorithm is is Hash Layers from Roller et al. (2021).”
Mixture-of-Experts with Expert Choice Routing (2022, methodology): “Hash layers [8, 28] devise hashing techniques on input tokens.”
Efficient Transformers: A Survey (2020, background): “However, we believe that the emergence of an entire research direction (Roller et al., 2021; Lewis et al., 2021; Lepikhin et al., 2020; Du et al., 2021) based on sparse efficient should warrant a new category of efficient Transformers.”
DEMix Layers: Disentangling Domains for Modular Language Modeling (2021, methodology): “This is an example of conditional computation (Fedus et al., 2021; Lepikhin et al., 2020; Lewis et al., 2021; Roller et al., 2021).”
A Survey of Transformers (2021, methodology): “As to using a learnable routing function for expert assignment, Roller et al. [110] design hash layers where tokens are hashed into a fixed number of buckets, each bucket corresponding to an expert.”
DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models (2024, unknown): “Hash Layer (Roller et al., 2021) and StableMoE (Dai et al., 2022) use fixed routing strategies for more stable routing and training.”
Dense-to-Sparse Gate for Mixture-of-Experts (2021, methodology): “The routing strategy deﬁned by the gate is key to MoE models, as it not only determines the model sparse structure (i.e., which experts tokens are routed to), but also affects the specialization of experts (i.e., how each expert is trained) (Roller et al., 2021).”
A Survey on Mixture of Experts in Large Language Models (2024, unknown): “This enables dynamic adjusting of layer-speciﬁc weights, extending the model versatility.”
Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints (2022, methodology): “A number of routing algorithms have been developed, for example Top-K (Shazeer et al., 2017), BASE and Sinkhorn-BASE layers (Lewis et al., 2021; Clark et al., 2022), Hash layers (Roller et al., 2021), and Expert Choice routing (Zhou et al., 2022).”
ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generation (2021, methodology): “These models [20, 26, 27, 28, 29] use mixture-of-experts to select different parameters for each incoming example.”
ST-MoE: Designing Stable and Transferable Sparse Expert Models (2022, methodology): “However, this leads to an interesting question of how architectures that eliminate learned routing Roller et al. (2021); Zuo et al. (2021) appear to perform well.”
A Survey on Dynamic Neural Networks for Natural Language Processing (2022, methodology): “They pro-pose a Switch Layer which only routes to one expert at a time, to reduce gating computation, batch size and communication costs.”
Designing Effective Sparse Expert Models (2022, methodology): “However, this leads to an interesting question of how architectures that eliminate learned routing Roller et al. (2021); Zuo et al. (2021) appear to perform well.”
Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts (2022, background): “Many works aim to improve the gating mechanism itself, by making it differentiable [38], reformulating as a linear assignment task [39] or even swapping it out for a simple hashing algorithm [40].”
From Sparse to Soft Mixtures of Experts (2023, background): “It has been observed that these mechanisms are often no better than random fixed routing (Roller et al., 2021).”
Tutel: Adaptive Mixture-of-Experts at Scale (2022, background): “First, it runs a gating function [19, 37, 43] that determines the destination GPU of each input token2 in the following all-to-all collective communication (All-to-All).”
QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Models (2023, background): “Those range from static assignment based on input tokens IDs (Roller et al., 2021), over dynamic token-to-expert matching (Zhou et al., 2022), to “soft” routing of linear input combinations (Puigcerver et al., 2023).”