End-To-End Multi-Task Learning With Attention

By Shikun Liu, Edward Johns, Andrew J. Davison. 2018

1,381 citations recorded by the source corpus.

We propose a novel multi-task learning architecture, which allows learning of task-specific feature-level attention. Our design, the Multi-Task Attention Network (MTAN), consists of a single shared network containing a global feature pool, together with a soft-attention module for each task. These modules allow for learning of task-specific features from the global features, whilst simultaneously allowing for features to be shared across different tasks. The architecture can be trained end-to-end and can be built upon any feed-forward neural network, is simple to implement, and is parameter efficient. We evaluate our approach on a variety of datasets, across both image-to-image predictions and image classification tasks. We show that our architecture is state-of-the-art in multi-task learning compared to existing methods, and is also less sensitive to various weighting schemes in the multi-task loss function. Code is available at https://github.com/lorenmt/mtan.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Neurology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Neurology

Citations in the corpus

Piggyback GAN: Efficient Lifelong Learning for Image Conditioned Generation (2021, background): “Multi-task attention networks [21] learn soft attention modules for each new task.”
Replay in Deep Learning: Current Approaches and Missing Biological Elements (2021, neutral): “…are closely related to the multitask learning literature, where the relationship between task similarity and network performance in terms of accuracy and time has been studied (Zhang & Yang, 2021; Ruder, 2017; Standley et al., 2020; Liu, Johns, & Davison, 2019; Kendall, Gal, & Cipolla, 2018).”
Reparameterizing Convolutions for Incremental Multi-Task Learning without Task Interference (2020, background): “On the one hand, encoder-focused approaches [41,25,36,10,43,33,1,61] emphasize learning feature representations from multi-task supervisory signals by employing architectures that encode shared and task-specific information.”
Which Tasks Should Be Learned Together in Multi-task Learning? (2019, background): “(Liu et al., 2019) adds a per-task attention layer after each layer, and (Maninis et al.”
Adaptive Scheduling for Multi-Task Learning (2019, methodology): “To train multi-task vision models, Liu et al. [12] propose a similar dynamic weight average approach.”
Learning to Predict User-Defined Types (2023, background): “New approaches include dynamically weighing tasks from gradients [57] and uncertainty [32], [33].”
FairMOT: On the Fairness of Detection and Re-identification in Multiple Object Tracking (2020, background): “Multi-task Learning There is a large body of literature (Liu et al. 2019; Kendall et al. 2018; Chen et al. 2018b; Guo et al. 2018; Sener and Koltun 2018) on multi-task learning which may be used to balance the object detection and reID feature extraction tasks.”
Collaboration by Competition: Self-coordinated Knowledge Amalgamation for Multi-talent Student Learning (2020, methodology): “The first is direct multi-task learning (dMTL) that intuitively train the two target task together under the same student network architecture but with labeled data as ground truth.”
Efficiently Identifying Task Groupings for Multi-Task Learning (2021, methodology): “In particular, dynamic loss reweighing has achieved performance superior to using ﬁxed loss weights found with extensive hyperparameter search [28, 18, 37, 10, 45, 33].”
MTI-Net: Multi-Scale Task Interaction Networks for Multi-Task Learning (2020, background): “Despite the progress reported by these or similar works [40, 28, 33, 26, 45], the joint learning of multiple tasks can lead to single-task performance degradation if information sharing happens between unrelated tasks.”
Multi-Task Learning with Deep Neural Networks: A Survey (2020, methodology): “Dynamic Weight Averaging (Liu et al., 2019) sets the task weights in the following way: λi(t) = N exp(ri(t− 1)/T )∑ j exp(rj(t− 1)/T ) where ri(t− 1) = Li(t− 1)/Li(t− 2) and T is a temperature hyperparameter.”
Multi-interactive Feature Learning and a Full-time Multi-modality Benchmark for Image Fusion and Segmentation (2023, background): “Learning from the dynamic weight average [45], we further present the factor of task preference η to emphasize the primary goal.”
AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning (2019, methodology): “Finally, we compare our method with state-of-the-art multi-task learning approaches, including Cross-Stitch Networks [39] (CVPR’16), Sluice Networks [48] (AAAI’19), and NDDR-CNN [16] (CVPR’19), which adopt several feature fusion layers between task-specific backbones, MTAN [33] (CVPR’19), which intr”
Conflict-Averse Gradient Descent for Multi-task Learning (2021, methodology): “MTAN originally experiments with equal loss weighting and two other dynamic loss weighting heuristics [15, 3].”
Blind Image Quality Assessment via Vision-Language Correspondence: A Multitask Learning Perspective (2023, methodology): “In our experiments, we follow the suggestions in [31], and compute wj(t − 1) through the moving average loss over the recent iterations in each epoch.”
Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scans (2021, background): “Sometimes, MTL papers take mix datasets for a “single” task and consider each dataset as a different task [37, 47, 35, 46].”
AuxNet: Auxiliary tasks enhanced Semantic Segmentation for Automated Driving (2019, methodology): “Joint learning of these tasks have shown significant performance gains in (Liu et al., 2018), (Eigen and Fergus, 2015), (Mousavian et al.”