Foundation Transformers

By Hongyu Wang, Shuming Ma, Shaohan Huang, Li Dong, Wenhui Wang, Zhiliang Peng, Yu Wu, Payal Bajaj, Saksham Singhal, Alon Benhaim and 5 more. 2022

30 citations recorded by the source corpus.

A big convergence of model architectures across language, vision, speech, and multimodal is emerging. However, under the same name"Transformers", the above areas use different implementations for better performance, e.g., Post-LayerNorm for BERT, and Pre-LayerNorm for GPT and vision Transformers. We call for the development of Foundation Transformer for true general-purpose modeling, which serves as a go-to architecture for various tasks and modalities with guaranteed training stability. In this work, we introduce a Transformer variant, named Magneto, to fulfill the goal. Specifically, we propose Sub-LayerNorm for good expressivity, and the initialization strategy theoretically derived from DeepNet for stable scaling up. Extensive experiments demonstrate its superior performance and better stability than the de facto Transformer variants designed for various applications, including language modeling (i.e., BERT, and GPT), machine translation, vision pretraining (i.e., BEiT), speech recognition, and multimodal pretraining (i.e., BEiT-3).

Subfields: Artificial Intelligence, Large Language Models, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Furu Wei
2. Li Dong
3. Shaohan Huang
4. Shuming Ma
5. Jianfeng Gao
6. Wenhui Wang
7. Mohit Bansal
8. Xinggang Wang

In Large Language Models

1. Furu Wei
2. Li Dong
3. Shaohan Huang
4. Shuming Ma
5. Wenhui Wang
6. Xia Song
7. Zewen Chi
8. Saksham Singhal

In Model Architectures

1. Furu Wei
2. Wenhui Wang
3. Li Dong
4. Shuming Ma
5. Shaohan Huang
6. Mostafa Dehghani
7. Neil Houlsby
8. Xinggang Wang

Citations in the corpus

• Language Is Not All You Need: Aligning Perception with Language Models (2023, methodology): “In the future, we would like to scale up K OSMOS -1 in terms of model size [13, 14, 60], and integrate the speech [12] capability into K OSMOS -1.”
• Dual PatchNorm (2023, methodology): “Additionally, we evaluate the LayerNorm placements in NormFormer (Shleifer et al., 2021) and Sub LayerNorm (Wang et al., 2022) which add additional LayerNorms within each of the self-attention and MLP layers in the transformer block.”
• Retentive Network: A Successor to Transformer for Large Language Models (2023, unknown): “As shown in Equation (8), the retention layer uses Sub-LayerNorm (Wang et al., 2022b) to normalize outputs.”
• A Survey on Vision--Language--Action Models for Embodied AI (2024, unknown): “For a more comprehensive comparison of human datasets, we refer interested readers to [217].”
• EVA-02: A Visual Representation for Neon Genesis (2023, methodology): “These include gated linear unit [37, 110] with sigmoid linear unit (SiLU) [56] / swich activation [99] ( SwiGLU ) as the feedfor-ward network, sub-LN [4, 122] as the normalization layer, and 2D rotary position embedding ( RoPE ) [113] for positional information injection.”
• Foundation Models Defining a New Era in Vision: A Survey and Outlook (2025, unknown): “The KOSMOS-1 model consists of a Magneto-based Large Language Model (LLM) [109] as a general interface and xPos [139] encoders to encode different modalities.”