Scaling Laws for Generative Mixed-Modal Language Models

By Armen Aghajanyan, L. Yu, Alexis Conneau, Wei-Ning Hsu, Karen Hambardzumyan, Susan Zhang, Stephen Roller, Omer Levy, Luke S. Zettlemoyer. 2023

159 citations recorded by the source corpus.

Generative language models define distributions over sequences of tokens that can represent essentially any combination of data modalities (e.g., any permutation of image tokens from VQ-VAEs, speech tokens from HuBERT, BPE tokens for language or code, and so on). To better understand the scaling properties of such mixed-modal models, we conducted over 250 experiments using seven different modalities and model sizes ranging from 8 million to 30 billion, trained on 5-100 billion tokens. We report new mixed-modal scaling laws that unify the contributions of individual modalities and the interactions between them. Specifically, we explicitly model the optimal synergy and competition due to data and model size as an additive term to previous uni-modal scaling laws. We also find four empirical phenomena observed during the training, such as emergent coordinate-ascent style training that naturally alternates between modalities, guidelines for selecting critical hyper-parameters, and connections between mixed-modal competition and training stability. Finally, we test our scaling law by training a 30B speech-text model, which significantly outperforms the corresponding unimodal models. Overall, our research provides valuable insights into the design and training of mixed-modal generative models, an important new class of unified models that have unique distributional properties.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Yejin Choi
2. Sean Welleck
3. Ximing Lu
4. Mohit Bansal
5. Xu Han
6. Michael Tschannen
7. Ning Ding
8. Eirikur Agustsson

Citations in the corpus

• Will we run out of data? Limits of LLM scaling based on human-generated data (2022, background): “For example, scaling experiments for transfer learning could be used to quantify the proximity or synergy between different distributions (Hernandez et al., 2021; Aghajanyan et al., 2023) and identify new datasets which can effectively expand the stock of data.”
• SemDeDup: Data-efficient learning at web-scale through semantic deduplication (2023, background): “Notably, many of these models appear never to converge, as test performance continues to increase even after 10s of passes through these massive datasets [11, 12].”
• Exploring AI Ethics of ChatGPT: A Diagnostic Analysis (2023, background): “The scaling laws that govern the development of language models, as documented in recent literature [84, 85], suggest that we can expect to encounter even more expansive models that incorporate multiple modalities in the near future.”
• A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPT (2023, unknown): “[260] conduct empirical analyses to validate the Hoffmann scaling law and propose an additional formula that explores the relationship between different training tasks in multimodal model training settings.”
• Emerging Properties in Unified Multimodal Pretraining (2025, unknown): “While several efforts [1, 11, 70] attempt to scale up their unified models, they are still trained predominantly on image-text paired data from standard image generation and understanding tasks.”
• Faith and Fate: Limits of Transformers on Compositionality (2023, background): “Scratchpad: dp[4] = max(input[4], 0) = max(2, 0) = 2 dp[3] = max(input[3], input[4], 0) = max(5, 2, 0) = 5 dp[2] = max(dp[3], input[2] + dp[4], 0) = max(5, 1 + 2, 0) = 5 dp[1] = max(dp[2], input[1] + dp[3], 0) = max(5, 2 + 5, 0) = 7 dp[0] = max(dp[1], input[0] + dp[2], 0) = max(7, 3 + 5, 0) = 8”
• Finite Scalar Quantization: VQ-VAE Made Simple (2023, background): “…VQ has become popular component in image (Bao et al., 2021; Li et al., 2023) and audio (Baevski et al., 2019) representation learning, and is a promising building block for the next generation of multimodal large language models (Aghajanyan et al., 2022; Kim et al., 2023; Aghajanyan et al., 2023).”
• MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies (2024, background): “Since the discovery of scaling law (Kaplan et al., 2020; Rae et al., 2021; Aghajanyan et al., 2023), scientifically and predictably (Achiam et al., 2023; Hu et al., 2023; Du et al., 2024) scaling up the LLMs has been pursued from diverse perspectives, especially for the pre-training stage.”
• Chameleon: Mixed-Modal Early-Fusion Foundation Models (2024, background): “Chameleon represents images, in addition to text, as a series of discrete tokens and takes advantage of the scaling properties of auto-regressive Transformers (Ramesh et al., 2021; Aghajanyan et al., 2022, 2023; Yu et al., 2023).”