UniT: Multimodal Multitask Learning with a Unified Transformer

By Ronghang Hu. 2021

368 citations recorded by the source corpus.

We propose UniT, a Unified Transformer model to simultaneously learn the most prominent tasks across different domains, ranging from object detection to natural language understanding and multimodal reasoning. Based on the transformer encoder-decoder architecture, our UniT model encodes each input modality with an encoder and makes predictions on each task with a shared decoder over the encoded input representations, followed by task-specific output heads. The entire model is jointly trained end-to-end with losses from each task. Compared to previous efforts on multi-task learning with transformers, we share the same model parameters across all tasks instead of separately fine-tuning task-specific models and handle a much higher variety of tasks across different domains. In our experiments, we learn 7 tasks jointly over 8 datasets, achieving strong performance on each task with significantly fewer parameters. Our code is available in MMF at https://mmf.sh.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

In Model Architectures

Citations in the corpus

Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks (2021, methodology): “[24] attempts to eliminate the need for object detectors by directly extracting features from the raw pixels with CNNs. [1, 22, 28, 56] take a further step by applying Transform-ers to raw image patches and word tokens.”
Generalized Decoding for Pixel, Image, and Language (2022, background): “In light of the versatility of transformers [72], we are now witnessing a growing interest in building general-purpose models that can learn from and be applied to a diverse set of vision and vision-language tasks, through multi-task learning [27, 32], sequential decoding [7, 54, 76, 85], or unified”
MDETR - Modulated Detection for End-to-End Multi-Modal Understanding (2021, background): “Some recent work also attempts to build multipurpose multi-modal architectures that are able to tackle a variety of vision-language [7] as well as pure language tasks in a single architecture [14].”
OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework (2022, background): “[59] and [60] unify tasks of different modalities by designing various task-speciﬁc layers.”
Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality (2022, methodology): “We evaluate various conﬁgurations of the following multimodal transformers: CLIP [56], FLAVA [68], LXMERT [76], UniT [35], UNITER [12], VILLA [23], VinVL [90], ViLT [40], VisualBERT [47] and ViLBERT [51].”
FindIt: Generalized Localization with Natural Language Queries (2022, methodology): “1 FindIt-DET is trained and tested on COCO 17’ to match the settings of [28,19].”
Omnivore: A Single Model for Many Visual Modalities (2022, methodology): “Recently, Transform-ers have been used to fuse multiple modalities: Transform-ers have been used to fuse features in vision-and-language tasks [2, 17, 37, 40, 49, 56, 83, 86] and video-and-audio tasks [64], video-and-image tasks [7], and even tasks that involve video, audio, and text [1].”
Qwen-VL: A Frontier Large Vision-Language Model with Versatile Abilities (2023, background): “, 2021a, 2022), especially in the development of multi-task generalist models (Hu and Singh, 2021; Singh et al., 2022; Zhu et al., 2022; Yu et al., 2022; Wang et al., 2022a; Lu et al., 2022a; Bai et al., 2022).”
Satlas: A Large-Scale, Multi-Task Dataset for Remote Sensing Image Understanding (2022, methodology): “SATNET is inspired by recent work that employ task-specific output heads [20, 27, 32], as well as methods that synthesize features across remote sensing image time series [21,25].”
CDTrans: Cross-domain Transformer for Unsupervised Domain Adaptation (2021, methodology): “For multi-modal based networks, there are several works (Tsai et al., 2019; Li et al., 2021e; Hu & Singh, 2021) that apply cross-attention for multi-modal feature fusion, which demonstrates that attention mechanism is powerful at distilling noise and feature alignment.”
Crossing the Format Boundary of Text and Boxes: Towards Unified Vision-Language Modeling (2021, methodology): “The other threads of work, including MDETR [24] and UniT [21], add task-speciﬁc heads on top of an object detector [7] to perform VL tasks.”
A Survey on Vision Transformer (2020, methodology): “Recently, a Unified Transformer (UniT) [100] model is proposed to cope with multi-modal multi-task learning, which can simultaneously handle multiple tasks across different domains,”
LAVT: Language-Aware Vision Transformer for Referring Image Segmentation (2021, background): “1 (a)) to learn more effective cross-modal alignments, which is in concurrence with Transformer’s overwhelming success in many other vision-language tasks [22,29,39,47].”
A Survey of Visual Transformers (2021, methodology): “UniT [150] follows the architecture of DETR and applies a wide range of task for uni-ﬁed Transformer pretraining via different task-speciﬁc output heads simultaneously.”
Beyond Self-Attention: External Attention Using Two Linear Layers for Visual Tasks (2021, methodology): “Subsequently, transformer methods have been successfully applied tomanyvisual tasks, including image classification [12], [31], [32], [33], object detection [34], lower-level vision [35], semantic segmentation [36], tracking [37], video instance segmentation [38], image generation [39],multimodal le”
GLIPv2: Unifying Localization and Vision-Language Understanding (2022, background): “As a result, such unified models see the marginal mutual benefit or even performance degradation [24] compared with task-specific models.”