Generating Images with Multimodal Language Models

By Jing Yu Koh, Daniel Fried, Ruslan Salakhutdinov. Neural Information Processing Systems, 2023

384 citations recorded by the source corpus.

We propose a method to fuse frozen text-only large language models (LLMs) with pre-trained image encoder and decoder models, by mapping between their embedding spaces. Our model demonstrates a wide suite of multimodal capabilities: image retrieval, novel image generation, and multimodal dialogue. Ours is the first approach capable of conditioning on arbitrarily interleaved image and text inputs to generate coherent image (and text) outputs. To achieve strong performance on image generation, we propose an efficient mapping network to ground the LLM to an off-the-shelf text-to-image generation model. This mapping network translates hidden representations of text into the embedding space of the visual models, enabling us to leverage the strong text representations of the LLM for visual outputs. Our approach outperforms baseline generation models on tasks with longer and more complex language. In addition to novel image generation, our model is also capable of image retrieval from a prespecified dataset, and decides whether to retrieve or generate at inference time. This is done with a learnt decision module which conditions on the hidden representations of the LLM. Our model exhibits a wider range of capabilities compared to prior multimodal language models. It can process image-and-text inputs, and produce retrieved images, generated images, and generated text -- outperforming non-LLM based generation models across several text-to-image tasks that measure context dependence.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/6fb5c0eff3696ef252aca9638e10176ecce7cecb

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning (2024, background): “MLLMs have also been extended to various applications, including fine-grained localization [61, 24] such as object detection [72], video understanding [71, 29, 8], and image generation [20, 49].”
Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook (2024, neutral): “This approach enables LLMs to process inputs other than text, as seen in various examples [41-44].”
Generative Multimodal Models are In-Context Learners (2023, methodology): “Additionally, we utilize CLIP-ViT-L, as in GILL[37], to compute the CLIP-I score for measuring image similarity.”
Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action (2023, background): “…difﬁculties, a line of recent works in building multimodal systems has leveraged pre-trained LLMs, with some augmenting with new modality encoders [5, 46, 119], some adding modality speciﬁc decoders [14, 96] and others leveraging the LLM’s capabilities to build modular frameworks [64, 166, 173].”
SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension (2023, background): “With the impressive success of Large language models (LLM) [1, 5, 4], recent studies work on generative Multimodal Large Language Models (MLLMs) [6, 7, 8, 9, 10, 11, 12, 13, 14, 18, 19, 20, 21] to improve multimodal comprehension and generation through utilizing the strong generality of LLMs.”
Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model (2024, methodology): “Many efforts have been made to combine these approaches, including extending a language model to use a diffusion model as a tool, either explicitly [Liu et al., 2023] or by grafting a pretrained diffusion model onto the language model [Dong et al., 2023, Koh et al., 2024].”