Fully-Adaptive Feature Sharing in Multi-Task Networks with Applications in Person Attribute Classification

By Yongxi Lu, Abhishek Kumar, Shuangfei Zhai, Yu Cheng, Tara Javidi, Rogério Schmidt Feris. 2016

407 citations recorded by the source corpus.

Multi-task learning aims to improve generalization performance of multiple prediction tasks by appropriately sharing relevant information across them. In the context of deep neural networks, this idea is often realized by hand-designed network architectures with layers that are shared across tasks and branches that encode task-specific features. However, the space of possible multi-task deep architectures is combinatorially large and often the final architecture is arrived at by manual exploration of this space, which can be both error-prone and tedious. We propose an automatic approach for designing compact multi-task deep learning architectures. Our approach starts with a thin multi-layer network and dynamically widens it in a greedy manner during training. By doing so iteratively, it creates a tree-like deep architecture, on which similar tasks reside in the same branch until at the top layers. Evaluation on person attributes classification tasks involving facial and clothing attributes suggests that the models produced by the proposed method are fast, compact and can closely match or exceed the state-of-the-art accuracy from strong baselines by much more expensive models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Exploring Shared Structures and Hierarchies for Multiple NLP Tasks (2018, neutral): “Recently, to break through the constraints of the shared structure, Lu et al. (2017) proposes a bottom-up approach that dynamically creates branches of networks during training that promotes grouping of similar tasks.”
Reparameterizing Convolutions for Incremental Multi-Task Learning without Task Interference (2020, background): “Unlike existing approaches [36,41] which learn such functions f(·) on the layer level of the network, i.”
Multi-task Deep Reinforcement Learning with PopArt (2018, background): “That the scaling issues considered in this paper, may have been a factor in the limited adoption is indicated by the wider use of this kind of learning in supervised settings (Johnson et al. 2017; Lu et al. 2016; Misra et al. 2016; Hashimoto et al. 2016), where loss functions are naturally well scal”
GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks (2017, background): “, 2014) and (Lu et al., 2016) use groupings amongst labels to search through possible architectures for learning.”
Which Tasks Should Be Learned Together in Multi-task Learning? (2019, background): “Examples include (Dai et al., 2016; Duong et al., 2015; Misra et al., 2016; Tessler et al., 2017; Yang & Hospedales, 2017; Lu et al., 2017).”
Beyond Shared Hierarchies: Deep Multitask Learning through Soft Layer Ordering (2017, methodology): “Such techniques have been applied across settings such as vision (Bilen and Vedaldi, 2016; 2017; Jou and Chang, 2016; Lu et al., 2017; Misra et al., 2016; Ranjan et al., 2016; Yang and Hospedales, 2017; Zhang et al., 2014), natural language (Collobert and Weston, 2008; Dong et al., 2015; Hashimoto…”
SpotTune: Transfer Learning Through Adaptive Fine-Tuning (2018, methodology): “Early methods were designed for shallow classification models [52, 17, 36], while more recent approaches address the problem of “with whom” each task should share features using deep neural networks [29, 32].”
Multi-task Learning based Pre-trained Language Model for Code Completion (2020, positive): “Multi-task learning has been successfully used in many fields including natural language processing [10, 14, 31], speech recognition [9] and computer vision [32, 34].”
Small Towers Make Big Differences (2020, background): “, 2018), adaptively and dynamically deciding which layers to share for which tasks during the training process (Lu et al., 2017; Vandenhende et al., 2019), or on a higher level, deciding which tasks should be learned together (Standley et al.”
Stochastic Filter Groups for Multi-Task CNNs: Learning Specialist and Generalist Convolution Kernels (2019, methodology): “The second group of MTL methods focuses on weight clustering based on task-similarity [33, 11, 14, 21, 23].”
MTI-Net: Multi-Scale Task Interaction Networks for Multi-Task Learning (2020, background): “Despite the progress reported by these or similar works [40, 28, 33, 26, 45], the joint learning of multiple tasks can lead to single-task performance degradation if information sharing happens between unrelated tasks.”
Evolutionary architecture search for deep multitask networks (2018, background): “DeepMTL has extended these ideas to domains where deep learning thrives, including vision [2, 17, 26, 31, 33, 35, 49, 51], speech [11, 12, 17, 39, 48], natural language processing [4, 6, 9, 17, 24, 27, 50], and reinforcement learning [5, 15, 47].”
A Modulation Module for Multi-task Learning with Applications in Image Retrieval (2018, background): “[17], who propose to automatically discover a neural network design to group similar tasks together; Yang et al.”
A Survey of Model Compression and Acceleration for Deep Neural Networks (2017, background): “Another example is the top face veriﬁcation results on the Labeled Faces in the Wild (LFW) dataset were obtained with networks containing hundreds of millions of parameters, using a mix of convolutional, locally-connected, and fully-connected layers [2], [3].”
Fashion IQ: A New Dataset Towards Retrieving Images by Natural Language Feedback (2021, methodology): “Visual attributes, including color, shape, and texture, have been successfully used to model clothing images [25, 22, 23, 2, 73, 7, 40].”
MTL-NAS: Task-Agnostic Neural Architecture Search Towards General-Purpose Multi-Task Learning (2020, methodology): “Our work leverages NAS to move towards GP-MTL, which (or whose search space) is compatible with any task combinations [27, 29,36–38,40,42, 52,67].”
AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning (2019, methodology): “Only a few methods have attempted to learn multibranch network architectures, using greedy optimization based on task affinity measures [34, 57] or convolutional filter grouping [6, 54].”