CM3: A Causal Masked Multimodal Model of the Internet

By Armen Aghajanyan, Po-Yao (Bernie) Huang, Candace Ross, Vladimir Karpukhin, Hu Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi, Gargi Ghosh, Michael Lewis and 1 more. 2022

180 citations recorded by the source corpus.

We introduce CM3, a family of causally masked generative models trained over a large corpus of structured multi-modal documents that can contain both text and image tokens. Our new causally masked approach generates tokens left to right while also masking out a small number of long token spans that are generated at the end of the string, instead of their original positions. The casual masking object provides a type of hybrid of the more common causal and masked language models, by enabling full generative modeling while also providing bidirectional context when generating the masked spans. We train causally masked language-image models on large-scale web and Wikipedia articles, where each document contains all of the text, hypertext markup, hyperlinks, and image tokens (from a VQVAE-GAN), provided in the order they appear in the original HTML source (before masking). The resulting CM3 models can generate rich structured, multi-modal outputs while conditioning on arbitrary masked document contexts, and thereby implicitly learn a wide range of text, image, and cross modal tasks. They can be prompted to recover, in a zero-shot fashion, the functionality of models such as DALL-E, GENRE, and HTLM. We set the new state-of-the-art in zero-shot summarization, entity linking, and entity disambiguation while maintaining competitive performance in the fine-tuning setting. We can generate images unconditionally, conditioned on text (like DALL-E) and do captioning all in a zero-shot setting with a single model.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Multimodal Learning

Citations in the corpus

InCoder: A Generative Model for Code Infilling and Synthesis (2022, methodology): “Most related to our approach is the work of Donahue et al. [23] and CM3 [2], who train left-to-right language models to fill in masked token segments of varying lengths; and the work of Alon et al. [7], who train an infilling-capable, AST-structured generative model of code on a smaller scale.”
Code Llama: Open Foundation Models for Code (2023, methodology): “We train in lling models following the concept of causal masking (Aghajanyan et al., 2022; Fried et al., 2023), where parts of a training sequence are moved to the end, and the reordered sequence is predicted autoregressively.”
Flamingo: a Visual Language Model for Few-Shot Learning (2022, background): “We emphasize few-shot learning and vision tasks while CM3 [1] primarily evaluates on language-only benchmarks in a zero-shot or fine-tuned setup.”
Language Is Not All You Need: Aligning Perception with Language Models (2023, background): “More importantly, unlocking multimodal input [2, 3, 4, 5, 6, 7] greatly widens the applications of language models to more high-value areas, such as multimodal machine learning, document intelligence, and robotics.”
MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation (2023, methodology): “InCoder [4] is a 6.7B parameter language model trained using a causal masking objective [30].”
DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation (2022, background): “More broadly, given that 1) infilling is an important functionality for real-world programming and 2) there is a growing trend in pre-training with the right context (Aghajanyan et al., 2022; Fried et al., 2022; Bavarian et al., 2022; Tay et al., 2022), we expect more future pre-trained models to pe”
Efficient Training of Language Models to Fill in the Middle (2022, positive): “Similar to our work, Zhu et al. [2019], Donahue et al. [2020], GLM [Du et al., 2022], CM3 [Aghajanyan et al., 2022], and InCoder [Fried et al., 2022] utilize left-to-right autoregressive modeling by moving the infill regions to the end of context, with regions separated by sentinels.”
CodeT: Code Generation with Generated Tests (2022, neutral): “INCODER is a unified generative model that can perform leftto-right code generation and code infilling (editing) via the causal mask language modelling training objective (Aghajanyan et al., 2022).”
Grounding Language Models to Images for Multimodal Generation (2023, methodology): “…settings are those which produce interleaved images and texts as outputs, which prior work (Tsimpoukelli et al., 2021; Alayrac et al., 2022) is incapable of, or does not generate semantically meaningful outputs for (see appendix for further comparison with CM3 (Aghajanyan et al., 2022)).”
OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models (2023, background): “Evaluations suggest that autoregressive vision-language models can be performant foundation models [5]: models like Flamingo [3], CM3 [1], Kosmos-1 [12], PALM-E [8], and multimodal GPT-4 [28] generalize well across diverse vision-language tasks. spacesuit.”
A Review on Language Models as Knowledge Bases (2022, methodology): “However, such a method would be expensive and impractical for modifying a few datapoints, especially for large LMs.”
Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models (2022, background): “Scaling Laws: We have consistently seen performance gains by scaling model size [5, 6, 7, 43, 44], and scale itself has been known to push internal model behavior away from classical bias-variance regimes [45].”
Finite Scalar Quantization: VQ-VAE Made Simple (2023, background): “…VQ has become popular component in image (Bao et al., 2021; Li et al., 2023) and audio (Baevski et al., 2019) representation learning, and is a promising building block for the next generation of multimodal large language models (Aghajanyan et al., 2022; Kim et al., 2023; Aghajanyan et al., 2023).”
Automated Program Repair in the Era of Large Pre-trained Language Models (2022, unknown): “Recently, researchers have also combined MLM with generative models to perform both bidirectional and autoregressive text generation or inﬁlling [45].”
Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook (2024, unknown): “For example, the causal masked multimodal (CMM) model [192] produces HTML markup that can be transformed into rich web pages with text, formatting, links, and images.”
Retrieval-Augmented Multimodal Language Modeling (2023, methodology): “Recently, CM3 (Aghajanyan et al., 2022) provides a unified model that uses a Transformer to perform both text and image generation.”