Language Models Can See: Plugging Visual Controls in Text Generation

By Yixuan Su, Tian Lan, Yahui Liu, Fangyu Liu, Dani Yogatama, Yan Wang, Lingpeng Kong, Nigel Collier. 2022

120 citations recorded by the source corpus.

Generative language models (LMs) such as GPT-2/3 can be prompted to generate text with remarkable quality. While they are designed for text-prompted generation, it remains an open question how the generation process could be guided by modalities beyond text such as images. In this work, we propose a training-free framework, called MAGIC (iMAge-Guided text generatIon with CLIP), for plugging in visual controls in the generation process and enabling LMs to perform multimodal tasks (e.g., image captioning) in a zero-shot manner. MAGIC is a simple yet efficient plug-and-play framework, which directly combines an off-the-shelf LM (i.e., GPT-2) and an image-text matching model (i.e., CLIP) for image-grounded text generation. During decoding, MAGIC influences the generation of the LM by introducing a CLIP-induced score, called magic score, which regularizes the generated result to be semantically related to a given image while being coherent to the previously generated context. Notably, the proposed decoding scheme does not involve any gradient update operation, therefore being computationally efficient. On the challenging task of zero-shot image captioning, MAGIC outperforms the state-of-the-art method by notable margins with a nearly 27 times decoding speedup. MAGIC is a flexible framework and is theoretically compatible with any text generation tasks that incorporate image grounding. In the experiments, we showcase that it is also capable of performing visually grounded story generation given both an image and a text prompt.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Andy Zeng
2. Tat-seng Chua
3. Dani Yogatama
4. Yan Wang
5. Lingpeng Kong
6. Michael S. Ryoo
7. Deng Cai
8. Vikas Sindhwani

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Luc Van Gool
3. Andrew Zisserman
4. Trevor Darrell
5. Jitendra Malik
6. Kristen Grauman
7. Yi Yang
8. Li Fei-Fei

In Natural Language Processing

1. Dani Yogatama
2. Yan Wang
3. Tat-seng Chua
4. Tian Lan
5. Deng Cai
6. Michael S. Ryoo
7. Krzysztof Marcin Choromanski
8. Vikas Sindhwani

In Multimodal Learning

1. Tat-seng Chua
2. Wei Ji
3. Tian Lan
4. Deng Cai
5. Dani Yogatama
6. Michael S. Ryoo
7. Leigang Qu
8. Andy Zeng

Citations in the corpus

• Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, background): “3, exceeding the performance of MAGIC (Su et al., 2022) which finetunes the text generator on the training set’s unpaired captions.”
• A Contrastive Framework for Neural Text Generation (2022, background): “In this work, we focus on studying the task of open-ended text generation due to its generality in various applications, such as story generation [7, 20], contextual text completion [18], poetry generation [14], and dialogue systems [22].”
• NExT-GPT: Any-to-Any Multimodal LLM (2023, background): “On the one hand, most of the researchers build fundamental MM-LLMs by aligning the well-trained encoders of various modalities to the textual feature space of LLMs, so as to let LLMs perceive other modal inputs [35, 110, 76, 40].”
• PandaGPT: One Model To Instruction-Follow Them All (2023, background): “Flamingo [1], BLIP-2 [11], and MAGIC [25] bridge powerful pre-trained vision-only and language-only models and show strong zero-shot abilities.”