DreamLLM: Synergistic Multimodal Comprehension and Creation

By Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jian‐Yuan Sun, Hongyu Zhou, Xiangwen Kong and 3 more. International Conference on Learning Representations, 2023

336 citations recorded by the source corpus.

This paper presents DreamLLM, a learning framework that first achieves versatile Multimodal Large Language Models (MLLMs) empowered with frequently overlooked synergy between multimodal comprehension and creation. DreamLLM operates on two fundamental principles. The first focuses on the generative modeling of both language and image posteriors by direct sampling in the raw multimodal space. This approach circumvents the limitations and information loss inherent to external feature extractors like CLIP, and a more thorough multimodal understanding is obtained. Second, DreamLLM fosters the generation of raw, interleaved documents, modeling both text and image contents, along with unstructured layouts. This allows DreamLLM to learn all conditional, marginal, and joint multimodal distributions effectively. As a result, DreamLLM is the first MLLM capable of generating free-form interleaved content. Comprehensive experiments highlight DreamLLM's superior performance as a zero-shot multimodal generalist, reaping from the enhanced learning synergy. Project page: https://dreamllm.github.io.

View Paper: https://www.semanticscholar.org/paper/7b689adb8c156d6158660f90d1c86888ee281f63

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Renrui Zhang
2. Peng Gao
3. Hongsheng Li
4. Jiaming Han
5. Michihiro Yasunaga
6. Chunting Zhou
7. Percy Liang
8. Bo Li

Citations in the corpus

• SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models (2023, methodology): “For more modality input, ImageBind-LLM (Han et al., 2023) and PandaGPT (Su et al., 2023) further incorporate audio and video conditions guided by ImageBind (Girdhar et al., 2023).”
• Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model (2024, methodology): “Many efforts have been made to combine these approaches, including extending a language model to use a diffusion model as a tool, either explicitly [Liu et al., 2023] or by grafting a pretrained diffusion model onto the language model [Dong et al., 2023, Koh et al., 2024].”
• 3D-VLA: A 3D Vision-Language-Action Generative World Model (2024, unknown): “…advances in various domains, including vision and language understanding (Li et al., 2022; 2023b; Liu et al., 2023; Huang et al., 2023c; Peng et al., 2023; Zhu et al., 2023), interleaved image and text understanding (Alayrac et al., 2022), interleaved image and text generation (Dong et al., 2023).”
• SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation (2024, unknown): “Some pioneering work [12, 13, 14, 15, 16, 17, 18, 19] further empower LLMs with the ability to generate images beyond texts.”
• CogVLM: Visual Expert for Pretrained Language Models (2023, unknown): “…the latest state-of-the-art generalist models, including MiniGPT-4 [Zhu et al., 2023], IDEFICS-Instruct [Laurençon et al., 2023], OpenFlamingo [Awadalla et al., 2023], DreamLLM [Dong et al., 2023], InstructBLIP [Dai et al., 2023], Fuyu [Bavishi et al., 2023], Qwen-VL [Bai et al., 2023], LLaVA-1.”
• Emu3: Next-Token Prediction is All You Need (2024, background): “There have been early efforts to unify vision understanding and generation [82, 97, 25, 21], exploring various generative objectives on image and text data.”
• Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction (2024, unknown): “Propelled by the principles of scaling laws in neural language models [30], several Large Language Models [9, 76, 70, 27, 63, 64] have been proposed, embodying the principle that increasing the scale of models tends to yield enhanced performance outcomes.”
• Emerging Properties in Unified Multimodal Pretraining (2025, unknown): “Unlike some prior approaches [18, 57, 69, 73] that introduce bottleneck connectors between generation and understanding modules, our design enables long-context interaction between multimodal understanding and generation through shared self-attention operations.”
• Show-o: One Single Transformer to Unify Multimodal Understanding and Generation (2024, unknown): “The flower in the middle has a few small yellow flowers.”
• CogAgent: A Visual Language Model for GUI Agents (2023, unknown): “…our model attained a score of 85.9, demonstrating superior handling of Vicuna-13B 22.4 - Otter [17] MPT-7B 24.7 - MiniGPT4 [44] Vicuna-13B 24.4 70.4 InstructBLIP [9] Vicuna-13B 25.6 77.3 LLaVA [21] LLaMA2-7B 28.1 66.3 LLaMA-Adapter v2 [14] LLaMA-7B 31.4 - DreamLLM [11] Vicuna-7B 35.9 76.5 LLaVA-1.”
• NExT-GPT: Any-to-Any Multimodal LLM (2023, background): “Lately, CoDi [78] has made strides in implementing the capability of simultaneously processing and generating arbitrary combinations of modalities, while it lacks the reasoning and decision-making prowess of LLMs as its core, and also is limited to the simple paired content generation.”
• InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model (2024, background): “…improve the MLLM in different dimensions, such as instruction tuning [11, 49, 98], efficient fine-tuning [33], high-resolution image inputs [6, 82, 83], hallucination mitigation [34, 87, 99], image generation [23, 30, 74, 89], 3D understanding [63] and image-text comprehension and composition [95].”
• Generative Multimodal Models are In-Context Learners (2023, background): “There are early studies on training more unified large multimodal models [23, 27, 71, 86] that are capable of performing visual understanding and generation simultaneously.”