Beyond Shared Hierarchies: Deep Multitask Learning through Soft Layer Ordering

By Elliot Meyerson, Risto Miikkulainen. 2017

98 citations recorded by the source corpus.

Existing deep multitask learning (MTL) approaches align layers shared between tasks in a parallel ordering. Such an organization significantly constricts the types of shared structure that can be learned. The necessity of parallel ordering for deep MTL is first tested by comparing it with permuted ordering of shared layers. The results indicate that a flexible ordering can enable more effective sharing, thus motivating the development of a soft ordering approach, which learns how shared layers are applied in different ways for different tasks. Deep MTL with soft ordering outperforms parallel ordering methods across a series of domains. These results suggest that the power of deep MTL comes from learning highly general building blocks that can be assembled to meet the demands of each task.

Subfields: Artificial Intelligence, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Meta & Continual Learning

Citations in the corpus

Exploring Shared Structures and Hierarchies for Multiple NLP Tasks (2018, neutral): “Meyerson and Miikkulainen (2017) introduced as a method for learning how to apply layers in different ways at different depths for different tasks, while simultaneously learning the layers themselves.”
Continual Learning via Local Module Composition (2021, background): “Modularity in neural networks is studied in the context of scalability [9], and more recently as a way to achieve compositionality and systematic generalization [6, 47, 15, 8, 27, 19] as well as for multi-task learning [65, 82].”
Recent Advances of Continual Learning in Computer Vision: An Overview (2021, positive): “The framework consists of several components including linear models, soft layer ordering [229], and soft gating as a modified version of soft layer ordering, which can be combined together in different orders to construct different models for different tasks.”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, neutral): “…Chen, Abbeel, and Schulman, 2017; Fernando, Banarse, Blundell, Zwols, Ha, Rusu, Pritzel, and Wierstra, 2017; Rosenbaum, Klinger, and Riemer, 2017; Meyerson and Miikkulainen, 2017; Kirsch, Kunze, and Barber, 2018; Ramachandran and Le, 2018; Chang, Gupta, Levine, and Griffiths, 2018; Liang,…”
Modular meta-learning (2018, methodology): “Meyerson and Miikkulainen [21] and later Liang et al. [22] expanded these ideas to the multitask setting with two particular compositional schemes: soft layer ordering and routing in DAGs.”
Lifelong Learning of Compositional Structures (2020, methodology): “…include treating the component selection as a reinforcement learning [26, 6, 11, 22], learning it via stochastic search [7, 2], or deﬁning it as a differentiable gating network [28], probabilistic model [13], or softmax selector [19] and training it jointly with the components via backpropagation.”
SpotTune: Transfer Learning Through Adaptive Fine-Tuning (2018, methodology): “Early methods were designed for shallow classification models [52, 17, 36], while more recent approaches address the problem of “with whom” each task should share features using deep neural networks [29, 32].”
Evolutionary architecture search for deep multitask networks (2018, methodology): “These methods have been reviewed in more detail in previous work [29, 37].”
Generalization in multitask deep neural classifiers: a statistical physics approach (2019, background): “Many recent studies have sought to explore alternative methods of parameter sharing, though these do not usually lend themselves as easily to this form of theoretical analysis [17, 18].”
How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Composition (2022, methodology): “, 2020c), such as vision (Rosenbaum et al., 2018; Meyerson and Miikkulainen, 2018; 2019) and robotics (Alet et al.”
AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning (2019, methodology): “In contrast, our approach allows learning of much more flexible architectures beyond tree-like structures, which have proven effective in multi-task learning [35], and relies on a more efficient, end-to-end learning method based on the final task performance, instead of greedy search based on task a”
Task-Driven Modular Networks for Zero-Shot Compositional Learning (2019, methodology): “These networks have been used in the past to speed up computation at test time [1] and to improve generalization for multi-task learning [19, 26], reinforcement learning [7], continual learning [30], visual question answering [2, 25],”
Dynamic Task Prioritization for Multitask Learning (2018, background): “Multitask learning benefits from multiple related tasks [63, 64] as they can reinforce one another and improve overall performance [23, 65].”
Gumbel-Matrix Routing for Flexible Multi-task Learning (2019, methodology): “These alphabets are the first 20 in the order originally reported by (Meyerson & Miikkulainen, 2018).”
Evolutionary neural AutoML for deep learning (2019, background): “The foundation is the soft-ordering multitask architecture [34] where each level of a fixed linear DNN consists of a fixed number of modules.”
Multi-Task Learning with Deep Neural Networks: A Survey (2020, methodology): “This architecture is also related to Soft Layer Ordering (Meyerson and Miikkulainen, 2017) (see section 2.”