One Model To Learn Them All

By Lukasz Kaiser, Aidan N. Gomez, Noam Shazeer, Ashish Vaswani, Niki Parmar, Llion Jones, Jakob Uszkoreit. 2017

350 citations recorded by the source corpus.

Deep learning yields great results across many fields, from speech recognition, image classification, to translation. But for each problem, getting a deep model to work well involves research into the architecture and a long period of tuning. We present a single model that yields good results on a number of problems spanning multiple domains. In particular, this single model is trained concurrently on ImageNet, multiple translation tasks, image captioning (COCO dataset), a speech recognition corpus, and an English parsing task. Our model architecture incorporates building blocks from multiple domains. It contains convolutional layers, an attention mechanism, and sparsely-gated layers. Each of these computational blocks is crucial for a subset of the tasks we train on. Interestingly, even if a block is not crucial for a task, we observe that adding it never hurts performance and in most cases improves it on all tasks. We also show that tasks with less data benefit largely from joint training with other tasks, while performance on large tasks degrades only slightly if at all.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

The Natural Language Decathlon: Multitask Learning as Question Answering (2018, background): “These tasks can also be learned with image classification and speech recognition with careful modularization [Kaiser et al., 2017], and the success of this approach extends to visual and textual question answering [Xiong et al., 2016].”
CTRL: A Conditional Transformer Language Model for Controllable Generation (2019, background): “…to extend proposals of uniﬁed, multi-task systems for all of NLP (Collobert & Weston, 2008; Collobert et al., 2011), parsing and tagging (Hashimoto et al., 2016), multiple languages (Wu et al., 2016; Johnson et al., 2017), and multiple modalities (Luong et al., 2015; Kaiser et al., 2017).”
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, methodology): “Finally, while scaling up existing approaches is one way to improve model quality, some approaches or architectures are more efﬁ-cient to train (Shazeer et al., 2017; Vaswani et al., 2017; Wu et al., 2019), more sample efﬁcient, or faster at inference (Gu et al., 2017; Roy et al., 2018).”
Self-Supervised MultiModal Versatile Networks (2020, background): “Using audio, vision and language to learn representations has also been explored in past work [11, 31, 38, 50, 83].”
Perceiver IO: A General Architecture for Structured Inputs & Outputs (2021, background): “[2, 3, 38]), but typically across a fixed and predefined set of modalities by means of domain-specific networks.”
Perceiver: General Perception with Iterative Attention (2021, methodology): “In other words, under these conditions, networks that use 2D convolutions cannot exploit the local neighorhood structure of inputs to reason about space, but must reason in terms of features, just like structure-agnostic architectures like Transformers and Per-ceivers.”
Measuring the Intrinsic Dimension of Objective Landscapes (2018, methodology): “Finally, as the ﬁeld departs from single stack-of-layers image classiﬁcation models toward larger and more heterogeneous networks (Ren et al., 2015; Kaiser et al., 2017) often composed of many modules and trained by many losses, methods like measuring intrinsic dimension that allow some automatic…”
Distributed Prioritized Experience Replay (2018, background): “A broad trend in deep learning is that combining more computation (Dean et al., 2012) with more powerful models (Kaiser et al., 2017) and larger datasets (Deng et al., 2009) yields more impressive results.”
Pretrained Transformers as Universal Computation Engines (2021, background): “Work specifically tackling multimodal tasks include (Kaiser et al. 2017), who showed a single model could learn a variety of multimodal tasks with an attention architecture.”
OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework (2022, background): “Uniﬁed Frameworks To pursue the uniﬁed models, [55] demonstrate a uniform format to represent tasks.”
A Generalist Agent (2022, background): “Kaiser et al. (2017) developed the MultiModel that trains jointly on 8 distinct speech, image and text processing tasks including classification, image captioning and translation.”
Soft Layer-Specific Multi-Task Summarization with Entailment and Question Generation (2018, background): “Several recent works have adopted MTL in neural models (Luong et al., 2016; Misra et al., 2016; Hashimoto et al., 2017; Pasunuru and Bansal, 2017; Ruder et al., 2017; Kaiser et al., 2017).”
Multi-Task Learning as Multi-Objective Optimization (2018, background): “…processing (Collobert and Weston, 2008; Dong et al., 2015; Liu et al., 2015a; Luong et al., 2015; Hashimoto et al., 2017), speech processing (Huang et al., 2013; Seltzer and Droppo, 2013; Huang et al., 2015), and even seemingly unrelated domains over multiple modalities (Kaiser et al., 2017).”
Dual Dynamic Inference: Enabling More Efficient, Adaptive, and Controllable Deep Inference (2019, background): “This mismatch between the limited resources of IoT devices and the high complexity of CNNs is only getting worse because the network architectures are getting more complex as they are designed to solve harder and larger-scale tasks [7].”
Multi-Task Learning with Deep Neural Networks: A Survey (2020, methodology): “Instead, (Kaiser et al., 2017) and (Pramanik et al., 2019) focus on building a “universal multi-modal multi-task model", in which a single model can handle multiple tasks with varying input domains.”
Scheduled Multi-Task Learning: From Syntax to Translation (2018, background): “…works have shown that parameter sharing between multiple sequence to sequence models that aim to solve different tasks may improve the accuracy of the individual tasks (Kaiser et al., 2017; Luong et al., 2015a; Zoph and Knight, 2016; Niehues and Cho, 2017; Bingel and Søgaard, 2017, inter-alia).”
Dynamic Task Prioritization for Multitask Learning (2018, background): “Some works [21] train a single model across multiple input domain modalities.”