NExT-GPT: Any-to-Any Multimodal LLM

By Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, Tat-seng Chua. International Conference on Machine Learning, 2023

842 citations recorded by the source corpus.

While recently Multimodal Large Language Models (MM-LLMs) have made exciting strides, they mostly fall prey to the limitation of only input-side multimodal understanding, without the ability to produce content in multiple modalities. As we humans always perceive the world and communicate with people through various modalities, developing any-to-any MM-LLMs capable of accepting and delivering content in any modality becomes essential to human-level AI. To fill the gap, we present an end-to-end general-purpose any-to-any MM-LLM system, NExT-GPT. We connect an LLM with multimodal adaptors and different diffusion decoders, enabling NExT-GPT to perceive inputs and generate outputs in arbitrary combinations of text, images, videos, and audio. By leveraging the existing well-trained highly-performing encoders and decoders, NExT-GPT is tuned with only a small amount of parameter (1%) of certain projection layers, which not only benefits low-cost training and also facilitates convenient expansion to more potential modalities. Moreover, we introduce a modality-switching instruction tuning (MosIT) and manually curate a high-quality dataset for MosIT, based on which NExT-GPT is empowered with complex cross-modal semantic understanding and content generation. Overall, our research showcases the promising possibility of building an AI agent capable of modeling universal modalities, paving the way for more human-like AI research in the community. Project page: https://next-gpt.github.io/

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/fa75a55760e6ea49b39b83cb85c99a22e1088254

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Large Language Models: A Survey (2024, positive): “There are already several prominent multi-modal LLMs out there, including: LLAVA [214], LLAVA-Plus [215], GPT-4 [33], Qwen-vl [116], Next-GPT [216], but the trend is expected to be continued.”
Show-o: One Single Transformer to Unify Multimodal Understanding and Generation (2024, methodology): “This demonstrates the great potential of our framework to unify multimodal understanding and generation in one single transformer. ii) When comparing with understanding only models including InstructBLIP (Dai et al., 2023), Qwen-VL-Chat (Bai et al., 2023), and mPLUG-Owl2 (Ye et al., 2024b) on multim”
A survey on multimodal large language models (2023, neutral): “The image is from [16].”
SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation (2024, positive): “Some pioneering work [12, 13, 14, 15, 16, 17, 18, 19] further empower LLMs with the ability to generate images beyond texts.”
Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook (2024, positive): “For instance, NExT-GPT [136] enhances an LLM with multimodal adapters and a range of diffusion decoders, enabling the model to process and generate outputs in any combination of text, images, videos, and audio.”
R1-VL: Learning to Reason with Multimodal Large Language Models via Step-Wise Group Relative Policy Optimization (2025, neutral): “For example, recent models [23, 39] incorporate multimodal inputs and outputs such as video, audio, and point cloud inputs beyond text and images.”
OmniGen: Unified Image Generation (2024, neutral): “Some works pro-pose integrating LLMs with diffusion models to equip LLMs with image generation capability [18, 60, 61, 69].”
Deep Multimodal Data Fusion (2024, methodology): “…Uni-Transformers Video + Audio AC X-llm [22] 2023 Multi-Transformers Image + Text VQA,VC MIST [43] 2023 Uni-Transformers Video + Text VQA NExT-GPT [199] 2023 Uni-Transformers Video + Text VQA,VC, VCR SA: sentiment analysis, ITR: image-text retrieval, VCR: visual commonsense reasoning, AS: action…”
Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding (2023, methodology): “Recent works [9, 61] have explored the use of separately pre-trained image and video encoders for processing, but these methods introduce model redundancy and prove challenging to train together.”
Intern VL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks (2023, background): “Recent advancements have seen the creation of vision large language models (VLLMs) [3, 21, 56, 59, 62, 66, 100, 121, 128, 130, 136, 138, 141, 142, 148], which aim to enhance language models with the capability to process and interpret visual information.”
Large Language Models for Education: A survey and outlook (2024, neutral): “Summarize.tech is a ChatGPT-powered tool that can summarize any long YouTube video, such as a lecture, a live event, or a government meeting.”
How far are we to GPT-4V? Closing the gap to commercial multimodal models with open-source suites (2024, neutral): “The development of open-source MLLMs [4, 15, 20, 43, 47, 49, 52, 65, 66, 98, 105, 113, 115, 119, 133, 134] has significantly influenced the AGI landscape by integrating and enhancing capabilities in processing both visual and textual data.”
Hunyuan-DiT: A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understanding (2024, background): “Various works have made efforts to equip text-to-image models with the multi-turn ability using MLLMs, such as Next-GPT [35], SEED-LLaMA [11], RPG [38], and DALLE-3 [5].”
Empowering Biomedical Discovery with AI Agents (2024, positive): “Alternatively, through representation alignment, data from diﬀerent modalities are analyzed by modality-speciﬁc models to generate representations, such as using the visual encoder from CLIP [122] for visual information processing.”