Language Models are General-Purpose Interfaces

By Yaru Hao, Haoyu Song, Li Dong, Shaohan Huang, Zewen Chi, Wenhui Wang, Shuming Ma, Furu Wei. 2022

113 citations recorded by the source corpus.

Foundation models have received much attention due to their effectiveness across a broad range of downstream applications. Though there is a big convergence in terms of architecture, most pretrained models are typically still developed for specific tasks or modalities. In this work, we propose to use language models as a general-purpose interface to various foundation models. A collection of pretrained encoders perceive diverse modalities (such as vision, and language), and they dock with a language model that plays the role of a universal task layer. We propose a semi-causal language modeling objective to jointly pretrain the interface and the modular encoders. We subsume the advantages and capabilities from both causal and non-causal modeling, thereby combining the best of two worlds. Specifically, the proposed method not only inherits the capabilities of in-context learning and open-ended generation from causal language modeling, but also is conducive to finetuning because of the bidirectional encoders. More importantly, our approach seamlessly unlocks the combinations of the above capabilities, e.g., enabling in-context learning or instruction following with finetuned encoders. Experimental results across various language-only and vision-language benchmarks show that our model outperforms or is competitive with specialized models on finetuning, zero-shot generalization, and few-shot learning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Multimodal Learning

Citations in the corpus

PaLM-E: An Embodied Multimodal Language Model (2023, background): “, 2020) models, recent years have seen a growing interest in large vision-language models (VLMs) (Li et al., 2019; Lu et al., 2019; Hao et al., 2022; Gan et al., 2022).”
A Survey on In-context Learning (2022, methodology): “Efforts were made to optimize prompting strategies with structured prompting (Hao et al., 2022b), demonstration ensembling (Khalifa et al., 2023), dynamic prompting (Zhou et al., 2023), and iterative forward tuning (Yang et al., 2023).”
Large Language Models: A Survey (2024, unknown): “…used for efficient developments of LLMs) includes Inner-Monologue [100], Megatron-Turing NLG [101], LongFormer [102], OPT-IML [103], MeTaLM [104], Dromedary [105], Palmyra [106], Camel [107], Yalm [108], MPT [109], ORCA-2 [110], Gorilla [67], PAL [111], Claude [112], CodeGen 2 [113],…”
OpenAGI: When LLM Meets Domain Experts (2023, methodology): “To transform these outputs into viable task planning solutions, we employ text similarity models to map them to the model name set, which is an established method in existing works [10].”
When large language models meet personalization: perspectives of challenges and opportunities (2023, unknown): “METALM[220] Pretrained Encoders with diverse modalities Transformer (pretrained from scratch) language-only tasks vision-language tasks”
DreamLLM: Synergistic Multimodal Comprehension and Creation (2023, background): “Another line of work instead explores training LLMs to consume and understand multimodal inputs (Hao et al., 2022; Huang et al., 2023; Chen et al., 2023b) with parameter-efficient tuning (Hu et al., 2022; Alayrac et al., 2022; Li et al., 2023d; Zhang et al., 2023e; Zhu et al., 2023a; Ye et al.,…”
Tool Learning with Foundation Models (2023, unknown): “In contrast, another solution is to building multimodal foundation models that can perceive general modalities, based on the belief that multimodal foundation models can all be unified through a general-purpose interface (Alayrac et al., 2022; Hao et al., 2022).”
MAPL: Parameter-Efficient Adaptation of Unimodal Pre-Trained Models for Vision-Language Few-Shot Prompting (2022, methodology): “Some methods (Dai et al., 2022; Hao et al., 2022) achieve this connection by finetuning the entire LM on image-text data, while others only train adapter layers inserted into the LM (Eichenberg et al.”
PointLLM: Empowering Large Language Models to Understand Point Clouds (2023, background): “These models have become versatile tools, acting as generalized interfaces [15] to perform an array of complex tasks [4, 38].”
Foundation Transformers (2022, methodology): “Following the previous work (Brown et al., 2020; Hao et al., 2022), we choose Winogrande (Sakaguchi et al.”
Foundation Models Defining a New Era in Vision: A Survey and Outlook (2025, unknown): “Hao et al. [107] develop MetaLM .”
Kosmos-2: Grounding Multimodal Large Language Models to the World (2023, background): “M ETA LM (Hao et al., 2022) 43.4 41.1 Flamingo-3B (Alayrac et al., 2022) 60.6 49.2 Flamingo-9B (Alayrac et al., 2022) 61.5 51.8 BLIP-2 (Vicuna-13B) (Li et al., 2023b) 71.6 65.0 K OSMOS -1 (Huang et al., 2023) 67.1 51.0 K OSMOS -2 (1.6B) 80.5 51.1 Table 4: Zero-shot image captioning results on the…”
LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks (2022, methodology): “This new language model is shown to work well on both language-only and vision-language tasks [84].”
Images Speak in Images: A Generalist Painter for In-Context Visual Learning (2022, background): “In NLP, the emergence of in-context learning [2, 7, 18] presents a new path in this direction, which uses language sequences as the general interface and allows the model to rapidly adapt to various language-centric tasks with only a handful of prompts and examples.”