PandaGPT: One Model To Instruction-Follow Them All

By Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, Deng Cai. Tsinghua Interdisciplinary Workshop on Logic, Language and Meaning, 2023

452 citations recorded by the source corpus.

We present PandaGPT, an approach to emPower large lANguage moDels with visual and Auditory instruction-following capabilities. Our pilot experiments show that PandaGPT can perform complex tasks such as detailed image description generation, writing stories inspired by videos, and answering questions about audios. More interestingly, PandaGPT can take multimodal inputs simultaneously and compose their semantics naturally. For example, PandaGPT can connect how objects look in an image/video and how they sound in an audio. To do so, PandaGPT combines the multimodal encoders from ImageBind and the large language models from Vicuna. Notably, only aligned image-text pairs are required for the training of PandaGPT. Thanks to the strong capability of ImageBind in embedding data from different modalities into the same space, PandaGPT displays emergent, i.e. zero-shot, cross-modal behaviors for data other than image and text (e.g., video, audio, depth, thermal, and IMU). We hope that PandaGPT serves as an initial step toward building AGI that can perceive and understand inputs in different modalities holistically, as we humans do.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/d3f79210b54e168c76b8c311488f42d7d1048b81

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models (2023, methodology): “For more modality input, ImageBind-LLM (Han et al., 2023) and PandaGPT (Su et al., 2023) further incorporate audio and video conditions guided by ImageBind (Girdhar et al., 2023).”
VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs (2024, methodology): “This includes comparisons with PandaGPT (Su et al., 2023), Video-LLaMA (Zhang et al., 2023) and MacawLLM (Lyu et al., 2023), which are adept at understanding complex multimodal scenes.”
Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic (2023, background): “Recently, interesting works have made remarkable progress by extending LLM to audio, e.g. , KOSMOS-1 (Huang et al., 2023), X-LLM (Chen et al., 2023), PandaGPT (Su et al., 2023) and control systems like PaLM-E (Driess et al., 2023) and EmbodiedGPT (Mu et al., 2023)”
SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension (2023, background): “With the impressive success of Large language models (LLM) [1, 5, 4], recent studies work on generative Multimodal Large Language Models (MLLMs) [6, 7, 8, 9, 10, 11, 12, 13, 14, 18, 19, 20, 21] to improve multimodal comprehension and generation through utilizing the strong generality of LLMs.”
SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models (2024, unknown): “As shown in Table 7, SPHINX-Plus, despite being an image-based model, significantly outperforms existing models (Jin et al., 2023; Su et al., 2023; Zhang et al., 2023a) specifically tailored for video tasks.”
Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook (2024, unknown): “A notable example is PandaGPT [50], which uniquely manages to interpret 6 different modalities at the same time, achieved through the integration of a multimodal encoder known as ImageBind [51].”
NExT-GPT: Any-to-Any Multimodal LLM (2023, methodology): “Profoundly, PandaGPT [77] achieves a comprehensive understanding of six different modalities simultaneously by integrating the multimodal encoder, i.”
PointLLM: Empowering Large Language Models to Understand Point Clouds (2023, background): “The second paradigm builds on pre-trained LLMs and uni-modal encoders, thereby avoiding training from scratch [1, 2, 7, 10, 11, 13, 23, 24, 27, 40, 54–56].”
SALMONN: Towards Generic Hearing Abilities for Large Language Models (2023, background): “…al., 2023a; Alayrac et al., 2022; Dai et al., 2023), silent video (Maaz et al., 2023; Chen et al., 2023b; Zhao et al., 2022), audio events (Gong et al., 2023b; Lyu et al., 2023) or speech (Chen et al., 2023a), or the encoders of multiple input types together (Su et al., 2023; Zhang et al., 2023b).”
Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding (2023, methodology): “PandaGPT (Su et al., 2023) utilizes multi-modal encoders from ImageBind, trained exclusively on image-instruction pairs, to enable large models to understand six modalities.”
VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use (2023, unknown): “In terms of Elo, the Human Verified GPT-4 reference achieves a higher rating than all alternatives, validating the quality of our reference set: concretely, for our Elo settings, the reference (Elo =1223) has an estimated win-rate over one of the best performing models, LLaVA, (Elo =1085) of 69%, an”
Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action (2023, methodology): “PandaGPT [170] and ImageBind-LLM [69] use the universal encoder ImageBind [56] to encode many kinds of input modalities, and ChatBridge [222] uses a similar universal encoder based on language.”
Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models (2023, unknown): “Furthermore, in addition to images, some studies have extended LLMs to other modalities such as audio (Wu et al., 2023; Su et al., 2023) and video (Maaz et al.”
A survey on multimodal large language models (2023, unknown): “Furthermore, some methods use a projection-based interface to close the modality gap [21,30,38,43].”
MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models (2023, unknown): “The evaluated MLLMs include BLIP-2 [30], LLaVA [33], MiniGPT-4 [56], mPLUG-Owl [50], LLaMA-Adapter-v2 [20], Otter [29], Multimodal-GPT [22], InstructBLIP [18], VisualGLM-6B [12], PandaGPT [42], ImageBind-LLM [2], and LaVIN [36].”
TempCompass: Do Video LLMs Really Understand Videos? (2024, methodology): “Recent Video LLMs (Su et al., 2023; Li et al., 2023c; Zhang et al., 2023; Lin et al., 2023a; Li et al., 2023d; Maaz et al., 2023; Luo et al., 2023; Li et al., 2023g) primarily follow the End-to-End paradigm, with optional temporal modules to model the temporal information across frames.”
A Survey on Vision--Language--Action Models for Embodied AI (2024, unknown): “PandaGPT [365] leverages ImageBind [231] to encode vision/text/audio/depth/thermal/IMU data, feeding them to the Vicuna model [108] also through a linear layer.”