Symbolic Discovery of Optimization Algorithms

By Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Yao Liu, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh and 2 more. 2023

679 citations recorded by the source corpus.

We present a method to formulate algorithm discovery as program search, and apply it to discover optimization algorithms for deep neural network training. We leverage efficient search techniques to explore an infinite and sparse program space. To bridge the large generalization gap between proxy and target tasks, we also introduce program selection and simplification strategies. Our method discovers a simple and effective optimization algorithm, $\textbf{Lion}$ ($\textit{Evo$\textbf{L}$ved S$\textbf{i}$gn M$\textbf{o}$me$\textbf{n}$tum}$). It is more memory-efficient than Adam as it only keeps track of the momentum. Different from adaptive optimizers, its update has the same magnitude for each parameter calculated through the sign operation. We compare Lion with widely used optimizers, such as Adam and Adafactor, for training a variety of models on different tasks. On image classification, Lion boosts the accuracy of ViT by up to 2% on ImageNet and saves up to 5x the pre-training compute on JFT. On vision-language contrastive learning, we achieve 88.3% $\textit{zero-shot}$ and 91.1% $\textit{fine-tuning}$ accuracy on ImageNet, surpassing the previous best results by 2% and 0.1%, respectively. On diffusion models, Lion outperforms Adam by achieving a better FID score and reducing the training compute by up to 2.3x. For autoregressive, masked language modeling, and fine-tuning, Lion exhibits a similar or better performance compared to Adam. Our analysis of Lion reveals that its performance gain grows with the training batch size. It also requires a smaller learning rate than Adam due to the larger norm of the update produced by the sign function. Additionally, we examine the limitations of Lion and identify scenarios where its improvements are small or not statistically significant. Lion is also successfully deployed in production systems such as Google search ads CTR model.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Stable and low-precision training for large-scale vision-language models (2023, methodology): “We examine CLIP-style models because of their importance in computer vision: CLIP-style models reach state-ofthe-art performance on a wide range of image classification tasks [44, 63, 42, 7] and underlie image generation methods such as DALLE·2 [47] and Stable Diffusion [49].”
AlphaEvolve: A coding agent for scientific and algorithmic discovery (2025, neutral): “In particular, classical evolutionary techniques have succeeded in symbolic regression applications [66, 87], automated scientific [21] or algorithmic [16] discovery, and scheduling [118] problems.”
Automated model building and protein identification in cryo-EM maps (2024, positive): “We improved the training of the model by using the Lion optimizer 53 and changing the dropout probability to 0.1 from 0 (ref.”
Mathematical discoveries from program search with large language models (2023, methodology): “Genetic programming is a subfield of computer science concerned with automatically generating or discovering computer programs using evolutionary methods 15,72,73 and is used for symbolic regression applications 74,75 and discovery of optimization algorithms 76 among others.”
DINOv2: Learning Robust Visual Features without Supervision (2023, neutral): “1%), obtained by Chen et al. (2023a).”
Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning (2023, methodology): “More methods have been introduced to enhance the efficiency of training LMs, such as dynamic architectures (Gong et al., 2019; Zhang & He, 2020) and efficient optimizers (Chen et al., 2023; Liu et al., 2023).”
Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training (2023, neutral): “On the other hand, Chen et al. (2023) automatically search among the light-weight gradient-based pre-conditioners and identify Lion, which is substantially faster than Adam on vision Transformers and diffusion models but achieves limited speed-up on LLMs (Chen et al., 2023).”
EVA-02: A Visual Representation for Neon Genesis (2023, background): “In Table 18, we show that sufficiently pre-trained EVA-02 representations are robust enough that can be fine-tuned using various numerical precisions ( e.g ., fp16 and bf16 ) and optimizers ( e.g ., Lion [25], AdamW [64, 84], and SGD [87]) .”