Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer

By Ge-feng Yang, Edward J. Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, N. Ryder, Jakub W. Pachocki, Weizhu Chen, Jianfeng Gao. 2021

177 citations recorded by the source corpus.

Subfields: Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

1. Joel Hestness

Citations in the corpus

• Training Compute-Optimal Large Language Models (2022, neutral): “[57] investigates how to choose a variety of these parameters for training an autoregressive transformer, including the learning rate and batch size.”
• The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only (2023, methodology): “Tasks Type Random small core main ext HellaSwag (Zellers et al., 2019) Sentence completion 25.0 ✓ ✓ ✓ ✓ LAMBADA (Paperno et al., 2016) Sentence completion 0.0 ✓ ✓ ✓ Winogrande (Sakaguchi et al., 2021) Coreference resolution 50.0 ✓ ✓ ✓ ✓ PIQA (Bisk et al., 2020) Multiple-choice question answering 50.”
• Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster (2023, methodology): “Further, when training models with a batch size smaller than the critical batch size, learning rate should be reduced linearly proportional to the reduction in batch size–consistent with the findings in (Shallue et al., 2018; Yang et al., 2021).”