Transformer is All You Need: Multimodal Multitask Learning with a Unified Transformer

By Ronghang Hu. 2021

51 citations recorded by the source corpus.

Subfields: Multimodal Learning, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Multimodal Learning

In Model Architectures

Citations in the corpus

Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, background): “2021)), or unifying model architectures (Hu and Singh 2021), Socratic Models1 (SMs) are a class of systems that embrace the zero-shot capabilities of foundation models by engineering guided discussions between the independent models to reach a shared consensus on a taskspecific output.”
SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers (2021, methodology): “Other works have also used Transformers in a variety of tasks such as tracking [61, 62], super-resolution [63], ReID [64], Colorization [65], Retrieval [66] and multi-modal learning [67, 68].”
Multiscale Vision Transformers (2021, background): “An emerging thread of work aims at applying transformers to vision tasks such as object detection [5], semantic segmentation [121, 105], 3D reconstruction [77], pose estimation [113], generative modeling [17], image retrieval [30], medical image segmentation [16,103,117], point clouds [45], video in”
What’s in Your Head? Emergent Behaviour in Multi-Task Transformer Models (2021, background): “The typical framework for training a model in natural language processing to perform multiple tasks is to have a shared pre-trained language model (LM), and add a small, compact neural network, often termed head, on top of the LM, for each task (Clark et al., 2019; Liu et al., 2019b; Nishida et al.,”
Towards General Purpose Vision Systems: An End-to-End Task-Agnostic Vision-Language Architecture (2021, background): “Latest systems like 12-in-1 [37] and UniT [19] have leveraged such representations to attempt to unify a number of different vision-language tasks, but the architectures still rely on task-speciﬁc heads.”
Does language help generalization in vision models? (2021, background): “Learning vision models using language supervision has gained popularity recently [18, 24, 4, 10, 17, 1, 8, 21].”
Effect of Vision-and-Language Extensions on Natural Language Understanding in Vision-and-Language Models (2021, background): “Some recent models, such as M6-v0 (Lin et al., 2021), UNIMO (Li et al., 2020), and UniT (Hu and Singh, 2021), have begun to focus on not only V&L tasks but also text-only tasks.”