Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer

By Greg Yang, J. Edward Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub W. Pachocki, Weizhu Chen, Jianfeng Gao. 2022

295 citations recorded by the source corpus.

Hyperparameter (HP) tuning in deep learning is an expensive process, prohibitively so for neural networks (NNs) with billions of parameters. We show that, in the recently discovered Maximal Update Parametrization (muP), many optimal HPs remain stable even as model size changes. This leads to a new HP tuning paradigm we call muTransfer: parametrize the target model in muP, tune the HP indirectly on a smaller model, and zero-shot transfer them to the full-sized model, i.e., without directly tuning the latter at all. We verify muTransfer on Transformer and ResNet. For example, 1) by transferring pretraining HPs from a model of 13M parameters, we outperform published numbers of BERT-large (350M parameters), with a total tuning cost equivalent to pretraining BERT-large once; 2) by transferring from 40M parameters, we outperform published numbers of the 6.7B GPT-3 model, with tuning cost only 7% of total pretraining cost. A Pytorch implementation of our technique can be found at github.com/microsoft/mup and installable via `pip install mup`.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Hardware and Architecture, Model Architectures, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Hardware and Architecture

In Model Architectures

In Scaling & Learning Theory

Citations in the corpus

GPT-NeoX-20B: An Open-Source Autoregressive Language Model (2022, neutral): “During the training of the model, however, new research has been released which may significantly alleviate the costs of hyperparameter tuning for future training runs [Yang et al., 2021].”
MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies (2024, methodology): “Details of the hyper-parameters and Tensor Program Operations are in Appendix A.1.”
Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone (2024, methodology): “We switched to GEGLU activation and used Maximal Update Parametrization (muP) [YHB + 22] to tune hyperparameters on a small proxy model and transfer them to the target 7B model.”
How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacy (2023, methodology): “Although some papers, such as (Yang et al., 2021), state that regularization and optimizer parameter values might not be transferable, Li et al. (2022b) reports success finding appropriate hyperparameters on smaller GPT models and reusing them for larger GPT models.”
Large Language Models (2023, methodology): “These ideas were also used to determine optimal modeldataset size tradeoff [58] and the scaling of hyperparameters [140].”
Effective Theory of Transformers at Initialization (2023, background): “In contrast, the Tensor Program V [25] puts forth a different scaling strategy – 1/C instead of 1/ √ C – based on heuristics that query and key vectors might get correlated at some point in training.”
Barren plateaus in variational quantum computing (2024, unknown): “Finally, as these are the most common BPs discussed in this Review, in what follows, we assume BPs to be probabilistic unless stated otherwise.”