Language Is Not All You Need: Aligning Perception with Language Models

By Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu and 7 more. 2023

786 citations recorded by the source corpus.

A big convergence of language, multimodal perception, action, and world modeling is a key step toward artificial general intelligence. In this work, we introduce Kosmos-1, a Multimodal Large Language Model (MLLM) that can perceive general modalities, learn in context (i.e., few-shot), and follow instructions (i.e., zero-shot). Specifically, we train Kosmos-1 from scratch on web-scale multimodal corpora, including arbitrarily interleaved text and images, image-caption pairs, and text data. We evaluate various settings, including zero-shot, few-shot, and multimodal chain-of-thought prompting, on a wide range of tasks without any gradient updates or finetuning. Experimental results show that Kosmos-1 achieves impressive performance on (i) language understanding, generation, and even OCR-free NLP (directly fed with document images), (ii) perception-language tasks, including multimodal dialogue, image captioning, visual question answering, and (iii) vision tasks, such as image recognition with descriptions (specifying classification via text instructions). We also show that MLLMs can benefit from cross-modal transfer, i.e., transfer knowledge from language to multimodal, and from multimodal to language. In addition, we introduce a dataset of Raven IQ test, which diagnoses the nonverbal reasoning capability of MLLMs.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Visual Instruction Tuning (2023, methodology): “Other LMMs trained on image-text pairs include BLIP-2 [28], FROMAGe [24], and KOSMOS-1 [20].”
LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention (2023, background): “The current working directory is /home/user/ echo “I am a chatbot“ > file.txt && cat file.txt && ls -l I am a chatbot -rw-r–r– 1 user user”
MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action (2023, background): “KOSMOS-1 [14] and PaLM-E [10] demonstrate the zeroshot multimodal CoT capabilities with large-scale training.”
ViperGPT: Visual Inference via Python Execution for Reasoning (2023, methodology): “Due to the practical difficulty of using these methods, the field has primarily moved towards end-to-end all-in-one models [1, 22, 23, 30].”
InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation (2023, methodology): “Previous research, such as Flamingo [7, 8], Kosmos-1 [63], and Multimodal C4 [64], conﬁrms that pretraining on the interleaved image-text sequences results in signiﬁcant multimodal in-context abilities.”
OBELISC: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents (2023, methodology): “Recent systems demonstrate the effectiveness of training large multimodal models such as Flamingo on naturally occurring multimodal documents (Alayrac et al., 2022; Aghajanyan et al., 2022; Huang et al., 2023).”
Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook (2024, neutral): “Furthermore, developing more efficient transformer architectures tailored for multimodal data, such as Kosmos-1 [162], Muse [163], and PixArt-α [164], presents a viable solution.”
DreamLLM: Synergistic Multimodal Comprehension and Creation (2023, methodology): “…is processed as patch embeddings with visual encoders H ϕ ( · ) like CLIP, which will then be encoded by a projector M ζ ( e.g. , a linear layer (Huang et al., 2023) or DETR-(Carion et al., 2020)/Perceiver-like (Jaegle et al., 2021) Resampler (Alayrac et al., 2022)) into L -length visual…”
Foundation Models Defining a New Era in Vision: A Survey and Outlook (2025, positive): “The model is trained on a combination of multi-modal corps from KOSMOS-1 [108] and GRIT for the next token prediction.”
Qwen-VL: A Frontier Large Vision-Language Model with Versatile Abilities (2023, background): “, 2023; Qwen, 2023), researchers have started building more powerful large vision-language models (LVLMs) based on LLMs (Alayrac et al., 2022; Chen et al., 2022; Li et al., 2023b; Dai et al., 2023; Huang et al., 2023; Peng et al., 2023; Zhu et al., 2023; Liu et al., 2023; Ye et al., 2023b,a; Chen et”
MotionGPT: Human Motion as a Foreign Language (2023, positive): “Recently, the wave of multi-modal models [20, 15, 19] is intriguing to process text along with other modalities, such as images [20, 15, 8], audio [13, 8], and videos [53].”
AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head (2023, background): “The rapid development of multi-modal LLMs (Wu et al., 2023; Shen et al., 2023; Huang et al., 2023b) has signiﬁcantly increased the research demand for evaluating its performance and behavior in understanding human intention, performing complex reasoning, and organizing the cooperation of multiple…”
Generative Multimodal Models are In-Context Learners (2023, background): “Recent years have witnessed the rapid growth of large multimodal models [5, 19, 31, 71].”
SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models (2023, background): “Flamingo (Alayrac et al., 2022) and Kosmos-1 (Huang et al., 2023) further exhibit promising multi-modal understanding performance for image-text interleaved contexts.”
PointLLM: Empowering Large Language Models to Understand Point Clouds (2023, methodology): “The first involves training the model from scratch, similar to text-only LLMs, us-ing large-scale multi-modal corpora and datasets [18, 36].”
mPLUG-OwI2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration (2023, neutral): “Instruction Tuning with MLLMs.”
LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models (2024, background): “Kosmos-1 [18] curates a 71M multimodal corpora, including arbitrarily interleaved documents.”