Scaling Vision Transformers to 22 Billion Parameters
By Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Steiner, Mathilde Caron, Robert Geirhos, Ibrahim M. Alabdulmohsin and 32 more. 2023
911 citations recorded by the source corpus.
The scaling of Transformers has driven breakthrough capabilities for language models. At present, the largest large language models (LLMs) contain upwards of 100B parameters. Vision Transformers (ViT) have introduced the same architecture to image and video modelling, but these have not yet been successfully scaled to nearly the same degree; the largest dense ViT contains 4B parameters (Chen et al., 2022). We present a recipe for highly efficient and stable training of a 22B-parameter ViT (ViT-22B) and perform a wide variety of experiments on the resulting model. When evaluated on downstream tasks (often with a lightweight linear model on frozen features), ViT-22B demonstrates increasing performance with scale. We further observe other interesting benefits of scale, including an improved tradeoff between fairness and performance, state-of-the-art alignment to human visual perception in terms of shape/texture bias, and improved robustness. ViT-22B demonstrates the potential for"LLM-like"scaling in vision, and provides key steps towards getting there.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Scaling & Learning Theory
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Mohit Bansal
- Jianfeng Gao
- Yoshua Bengio
- Quoc V. Le
- Percy Liang
- Luke S. Zettlemoyer
- Trevor Darrell
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Ming-Hsuan Yang
- Jitendra Malik
- Xiaogang Wang
- Andrew Zisserman
- Cordelia Schmid
In Scaling & Learning Theory
- Neil Houlsby
- Basil Mustafa
- Mostafa Dehghani
- Xiaohua Zhai
- Lucas Beyer
- Anurag Arnab
- Alexander Kolesnikov
- Matthias Minderer
Citations in the corpus
- Stable and low-precision training for large-scale vision-language models (2023, methodology): “Loss spikes can emerge when scaling up models [8, 25, 14, 69, 71, 54, 74].”
- MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action (2023, background): “Recent years have seen significant advancement for computer vision, thanks to improved network architecture [4, 9, 12,28], large-scale model training [8,29,35], and other factors.”
- PaLM-E: An Embodied Multimodal Language Model (2023, methodology): “2022), which we refer to as ViT-4B, and a similar 22 billion parameter model, ViT-22B (Dehghani et al., 2023), both of which have been pre-trained on image classification.”
- OtterHD: A High-Resolution Multi-modality Model (2023, neutral): “It also includes a layernorm for the Q and K embeddings before attention calculation [16].”
- Foundation models in robotics: Applications, challenges, and the future (2023, neutral): “The ViT transforms an image into a sequence of embedding vectors which are projected into the language embedding space via an affine transformation.”
- RALL-E: Robust Codec Language Modeling with Chain-of-Thought Prompting for Text-to-Speech Synthesis (2024, background): “Moreover, the simple yet effective next-token prediction task of LLMs makes it easy to apply LLMs on other fields, such as vision [5] and speech synthesis [29], as long as the data can be converted to discrete speech tokens.”
- PaLI-X: On Scaling up a Multilingual Vision and Language Model (2023, methodology): “Inspired by the work in [5], we present PaLI-X, a multilingual vision and language model with reusable scaled-up components, consisting of a pretrained large-capacity visual encoder (using [6] as the starting point) and a pretrained language-only encoder-decoder (using [7] as the starting point), fu”
- End-to-End Autonomous Driving: Challenges and Frontiers (2023, background): “In contrast, Transformer-based feature extractors [177], [178] show great scalability in perception tasks while not being widely adopted for end-to-end driving yet.”
- EVA-02: A Visual Representation for Neon Genesis (2023, methodology): “the study of state-of-the-art representations is frequently conducted using huge amounts of infrastructure and web-scale private training data [142, 3, 26, 38], which makes it difficult to evaluate the effects of modeling advancements in a feasible and transparent way, and restricts access to a broa”
- Scaling Open-Vocabulary Object Detection (2023, methodology): “Task performance improves predictably as training data and compute are increased [36], with recent work showing continued improvements for models with up to 22 billion parameters [5].”
- Genie: Generative Interactive Environments (2024, methodology): “Further, we employ bfloat16 and QK norm for training our dynamics model, which has been shown to stabilize training at large scale (Dehghani et al., 2023; Henry et al., 2020).”
- Kimi K2: Open Agentic Intelligence (2025, neutral): “On the other hand, Query-Key Normalization (QK-Norm) [12, 82] is not applicable to multi-head latent attention (MLA), because its Key matrices are not fully materialized during inference.”
- FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space (2025, positive): “To improve GPU utilization of single stream blocks, we leverage fused feed-forward blocks inspired by Dehghani et al. [8], which i) reduce the number of modulation parameters in a feedforward block by a factor of 2 and ii) fuse the attention input-and output linear layers with that of the MLP,…”
- Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, neutral): “Unlike the 6B version, the 0.3B variant employs standard LayerNorm [11] without QK-Norm [53].”
- Towards Generalist Biomedical AI (2023, background): “The emergence of the foundation model paradigm [5] has had widespread impact across a variety of applications in language [8], vision [15], and other modalities [16].”
- Chameleon: Mixed-Modal Early-Fusion Foundation Models (2024, methodology): “As inspired by Dehghani et al. (2023) and Wortsman et al. (2023), we first deviate from the Llama architecture by using query-key normalization (QK-Norm).”
- LongVILA: Scaling Long-Context Visual Language Models for Long Videos (2024, background): “…large models is typically a complex, systematic endeavor that demands both data engineering (Betker et al., 2023; Ouyang et al., 2022; Zhou et al., 2024) and system-software co-design (Lepikhin et al., 2020; Chowdhery et al., 2023; Shoeybi et al., 2019; Brown et al., 2020; Dehghani et al., 2023).”
- Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action (2023, background): “We observe extremely large values in the multi-head attention logits and find that applying Layer-Norm [10] to the queries and keys following [38] mitigates the problem.”
- On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspective (2023, background): “For instance, the recent ViT-22B [13] scales vision Transformer [14] to 22 billion parameters by training it on the 4 billion JFT dataset [58] (a larger version of the previous JFT-300M dataset [43]), which becomes the largest vision foundation model to date.”