Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework

By Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, Hongxia Yang. 2022

252 citations recorded by the source corpus.

Subfields: Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Model Architectures

Citations in the corpus

Flamingo: a Visual Language Model for Few-Shot Learning (2022, background): “[119] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang.”
VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts (2021, background): “Pre-training with Transformer [46] backbone networks has substantially advanced the state of the art across natural language processing [35, 11, 29, 23, 12, 37, 2, 8, 9, 5–7, 32], computer vision [13, 45, 3] and vision-language [44, 42, 4, 51, 36, 19, 21, 24, 47] tasks.”
BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models (2023, neutral): “…a rapid advancement in the past few years, where pre-trained models with increasingly larger scale have been developed to continuously push the state-of-the-art on various down-stream tasks (Radford et al., 2021; Li et al., 2021; 2022; Wang et al., 2022a; Alayrac et al., 2022; Wang et al., 2022b).”
CoCa: Contrastive Captioners are Image-Text Foundation Models (2022, methodology): “Another line of research [16, 17, 34] proposes encoder-decoder models trained with generative losses and shows strong results in vision-language benchmarks while the visual encoder still performs competitively on image classification.”
Kosmos-2: Grounding Multimodal Large Language Models to the World (2023, methodology): “OFA (Wang et al., 2022c) targets unifying various cross and mono modal tasks including image generation, visual grounding, image captioning, image classification, language modeling, etc., and has achieved impressive performances, by utilizing the concept of bounding boxes as tokens from Pix2Seq (Che”
A generalist vision–language foundation model for diverse biomedical tasks (2023, unknown): “We follow the fundamental principles of a unified model 51 : (1) modality-agnostic, (2) task-agnostic and (3) modality and task comprehensiveness.”
3D-LLM: Injecting the 3D World into Large Language Models (2023, methodology): “Augmenting LLM vocabularies with location tokens In order to align 3D spatial locations with LLMs, we propose to embed 3D locations in the vocabularies, following [6] and [43].”
A Generalist Framework for Panoptic Segmentation of Images and Videos (2022, methodology): “Our work is also related to recent generalist vision models [10, 11, 60, 31, 43] where both architecture and loss functions are task-agnostic.”
mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections (2022, methodology): “Following [4, 35], we concatenate the question with the object labels and OCR tokens extracted from image.”
Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning (2023, methodology): “Second, we design a pretraining strategy that effectively leverages cross-modal supervision in the form of transcribed speech from unlabeled narrated videos by reformulating sentence boundaries as pseudo event boundaries, as presented in Section 3.2 and Figure 3.”
DQ-DETR: Dual Query Detection Transformer for Phrase Extraction and Grounding (2022, methodology): “A simple way to extend existing REC models (Wang et al. 2022; Zhu et al. 2022) to PEG is to develop a two-stage solution: ﬁrstly extracting phrases using an NLP tool like spaCy (Honnibal and Montani 2017) and then applying a REC model.”
On the link between conscious function and general intelligence in humans and machines (2022, background): “…and compute are now enabling the development of architectures and task setups that can enable general-purpose, modality-and task-agnostic models (Jaegle et al., 2021; Wang et al., 2022), which could lay the foundations for agents that learn high-level concepts grounded in real world experience.”
Multimodal Learning With Transformers: A Survey (2022, background): “A large amount of recent attempts [117], [296], [297], [298] study how to use as unified as possible pipelines to handle various modalities and multimodal tasks.”
Black-box Prompt Learning for Pre-trained Language Models (2022, background): “The black-box prompt learning for large multi-modal models (Wang et al., 2021b; Singh et al., 2022; Wang et al., 2021c; Zhou et al., 2022; Wang et al., 2022; Diao et al., 2023) is another important scenario to explore in future work.”
Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone (2022, methodology): “Using our proposed two stage pre-training, FIBER achieves better performance than current SoTA [71] that uses a Large sized model.”