Continual and Multi-Task Architecture Search

By Ramakanth Pasunuru, Mohit Bansal. 2019

54 citations recorded by the source corpus.

Architecture search is the process of automatically learning the neural model or cell structure that best suits the given task. Recently, this approach has shown promising performance improvements (on language modeling and image classification) with reasonable training speed, using a weight sharing strategy called Efficient Neural Architecture Search (ENAS). In our work, we first introduce a novel continual architecture search (CAS) approach, so as to continually evolve the model parameters during the sequential training of several tasks, without losing performance on previously learned tasks (via block-sparsity and orthogonality constraints), thus enabling life-long learning. Next, we explore a multi-task architecture search (MAS) approach over ENAS for finding a unified, single cell structure that performs well across multiple tasks (via joint controller rewards), and hence allows more generalizable transfer of the cell structure knowledge to an unseen new task. We empirically show the effectiveness of our sequential continual learning and parallel multi-task learning based architecture search approaches on diverse sentence-pair classification tasks (GLUE) and multimodal-generation based video captioning tasks. Further, we present several ablations and analyses on the learned cell structures.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Meta & Continual Learning

In Neural Architecture Search & AutoML

Citations in the corpus

A Comprehensive Survey of Neural Architecture Search (2020, methodology): “Subsequently, CAS [54] explored a multi-task architecture search based on ENAS.”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, methodology): “This work is deeply related to RL methods for neural architecture search (Zoph and Le, 2016) and their one shot (Brock, Lim, Ritchie, and Weston, 2017), multi-task learning (Pasunuru and Bansal, 2019), and continual learning (Pasunuru and Bansal, 2019; Xu and Zhu, 2018) variants.”
Emergent Structures and Training Dynamics in Large Language Models (2022, background): “Pasunuru and Bansal (2019) Since attention-based models (Vaswani et al., 2017) have become state-of-the-art tools in NLP, there have been numerous attempts to provide some understanding of their predictions by visualizing the attentions layer.”