Chameleon: Mixed-Modal Early-Fusion Foundation Models

By Chameleon Team, Mingda Chen, Jacob Kahn, Shang-Wen Li. arXiv.org, 2024

918 citations recorded by the source corpus.

We present Chameleon, a family of early-fusion token-based mixed-modal models capable of understanding and generating images and text in any arbitrary sequence. We outline a stable training approach from inception, an alignment recipe, and an architectural parameterization tailored for the early-fusion, token-based, mixed-modal setting. The models are evaluated on a comprehensive range of tasks, including visual question answering, image captioning, text generation, image generation, and long-form mixed modal generation. Chameleon demonstrates broad and general capabilities, including state-of-the-art performance in image captioning tasks, outperforms Llama-2 in text-only tasks while being competitive with models such as Mixtral 8x7B and Gemini-Pro, and performs non-trivial image generation, all in a single model. It also matches or exceeds the performance of much larger models, including Gemini Pro and GPT-4V, according to human judgments on a new long-form mixed-modal generation evaluation, where either the prompt or outputs contain mixed sequences of both images and text. Chameleon marks a significant step forward in a unified modeling of full multimodal documents.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/32112b798f70faab00e14806f51d46058cf5e597

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Show-o: One Single Transformer to Unify Multimodal Understanding and Generation (2024, methodology): “In the example at the top of Fig.6, Chameleon, SEED-X, and Show-o all provide comprehensive descriptions of the image’s main content.”
SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation (2024, neutral): “Some work [15, 14, 13, 22, 23] perform multimodal autoregression with a unified next-word-prediction objective through using a discrete visual tokenizer.”
Emerging Properties in Unified Multimodal Pretraining (2025, neutral): “Typical design choices for unified multi-modal generation and understanding models include: Quantized AR. Autoregressive visual generation [11, 48, 59, 70, 79, 83–85, 90 ] Unfortunately, the visual generation quality of autoregressive models is empirically inferior to diffusion-based models.”
Emu3: Next-Token Prediction is All You Need (2024, methodology): “Despite early attempts at unifying generation and perception, such as Emu [82] and Chameleon [83], these efforts either resort to connecting LLMs with diffusion models or fail to match the performance of task-specific methods tailored for generation and perception.”
OmniGen: Unified Image Generation (2024, neutral): “Others use discrete tokens to support both image and text generation simultaneously [41, 62, 71].”
Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation (2024, neutral): “Unified multimodal understanding and generation models are considered powerful for facilitating seamless reasoning and generation across different modalities [74, 91].”
Infinity∞: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis (2024, positive): “Parti [79], Emu3 [70], chameleon[62] , Liquid[73], and VideoPoet[32] scaled up autoregressive models in text-to-image or video synthesis.”
Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation (2024, methodology): “Pioneering methods [Lu et al. 2022b, 2023; Bai et al. 2023b; Team et al. 2023; Team 2024] attempt to incorporate image generation into LLM using an autoregressive approach and achieve promising results.”
LongVILA: Scaling Long-Context Visual Language Models for Long Videos (2024, neutral): “There are two predominant designs for VLMs: the encoder-decoder architecture ( e.g., , LLaVA (Liu et al., 2023b), PaLM-E (Driess et al., 2023)) and the decoder-only architecture ( e.g., , Fuyu (Bavishi et al., 2023), Chameleon (Team, 2024)).”
SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics (2025, positive): “SmolVLA outperforms other VLA-based approaches such as Octo (Team et al., 2024) and OpenVLA (Kim et al., 2024), as well as the diffusion policy baseline across both LIBERO and Meta-World.”
Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models (2024, neutral): “Sketches are so fundamental that we use them to teach school children how to solve geometry problems by drawing support lines, to aid engineers conveying prototypes, to support architects creating blueprints, and to allow scientists like us to convey scientific contributions (see Figure 1).”
Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model (2024, methodology): “…multi-modal models includes examples such as Fuyu [Bavishi et al., 2023], which uses image patches as inputs for visual understanding, and Chameleon [Chameleon Team, 2024], which converts each image to a sequence of discretized tokens and then trains over the combined text-image token sequences.”