Grounding Language Models to Images for Multimodal Generation

By Jing Yu Koh, Ruslan Salakhutdinov, Daniel Fried. 2023

98 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Percy Liang
2. Quoc V. Le
3. Chunyuan Li
4. Philip S. Yu
5. Yong Jae Lee
6. Mohit Bansal
7. Jianfeng Gao
8. Ludwig Schmidt

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Dacheng Tao
3. Xiaogang Wang
4. Ming-Hsuan Yang
5. Yi Yang
6. Trevor Darrell
7. Yu Qiao
8. Jitendra Malik

In Large Language Models

1. Peng Jin
2. Haotian Liu
3. Yong Jae Lee
4. Song Han
5. Philip S. Yu
6. Ludwig Schmidt
7. William Yang Wang
8. Chunyuan Li

In Multimodal Learning

1. Peng Jin
2. Li Yuan
3. Chunyuan Li
4. Haotian Liu
5. Yong Jae Lee
6. Bin Lin
7. Munan Ning
8. Bin Zhu

Citations in the corpus

• Visual Instruction Tuning (2023, methodology): “Other LMMs trained on image-text pairs include BLIP-2 [28], FROMAGe [24], and KOSMOS-1 [20].”
• OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models (2023, background): “2 and LLaVa, can incorporate only one image in their context [6, 16, 22, 25, 39, 41], autoregressive vision-language models accept interleaved imagetext sequences, enabling in-context learning.”
• LayoutGPT: Compositional Visual Planning and Generation with Large Language Models (2023, background): “[23] proposes to learn translation parameters to map embeddings between visual and language domains such that an LLM can ground on both modalities.”
• MoE-LLaVA: Mixture of Experts for Large Vision-Language Models (2024, methodology): “…with strong instruction-following and generalization capabilities have been applied to LVLMs. Early works such as BLIP-2 (Li et al., 2023b) and FROMAGe (Koh et al., 2023) encoded visual signals into a sequence of visual tokens, successfully adapting vision to LLMs through several projection layers.”
• AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration (2023, unknown): “…al., 2022); VQA T : TextVQA (Singh et al., 2019); POPE (Li et al., 2023d); MME (Fu et al., 2023); MMB: MMBench (Liu et al., 2023b); MMB CN : MMBench-Chinese (Liu et al., 2023b); SEED: SEED-Bench (Li et al., 2023a); LLaVA W : LLaVA-Bench (In-the-Wild) (Liu et al., 2023a); MM-Vet (Yu et al., 2023).”
• A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPT (2023, methodology): “Recently, [145] proposed a method to realize VL dialogue with frozen language models, enabling the model to generate interleaved multimodal data.”
• Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic (2023, methodology): “Ad-ditionally, there are simpler and more direct meth-ods: FROMAGe (Koh et al., 2023) and LLaVA (Liu et al., 2023a) directly feed visual features to the LLM using only a learnable fully connected layer.”
• AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration (2023, background): “Large multi-modal models (LMMs) or visual language models (VLMs) are LLMs augmented with vision inputs [1, 22, 21, 11, 48, 26].”