A Modulation Module for Multi-task Learning with Applications in Image Retrieval

By Xiangyu Zhao, Haoxiang Li, Xiaohui Shen, Xiaodan Liang, Ying Wu. 2018

155 citations recorded by the source corpus.

Multi-task learning has been widely adopted in many computer vision tasks to improve overall computation efficiency or boost the performance of individual tasks, under the assumption that those tasks are correlated and complementary to each other. However, the relationships between the tasks are complicated in practice, especially when the number of involved tasks scales up. When two tasks are of weak relevance, they may compete or even distract each other during joint training of shared parameters, and as a consequence undermine the learning of all the tasks. This will raise destructive interference which decreases learning efficiency of shared parameters and lead to low quality loss local optimum w.r.t. shared parameters. To address the this problem, we propose a general modulation module, which can be inserted into any convolutional neural network architecture, to encourage the coupling and feature sharing of relevant tasks while disentangling the learning of irrelevant tasks with minor parameters addition. Equipped with this module, gradient directions from different tasks can be enforced to be consistent for those shared parameters, which benefits multi-task joint training. The module is end-to-end learnable without ad-hoc design for specific tasks, and can naturally handle many tasks at the same time. We apply our approach on two retrieval tasks, face retrieval on the CelebA dataset [12] and product retrieval on the UT-Zappos50K dataset [34, 35], and demonstrate its advantage over other multi-task learning methods in both accuracy and storage efficiency.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Reparameterizing Convolutions for Incremental Multi-Task Learning without Task Interference (2020, methodology): “Several works proposed to dynamically re-weight the loss function of each task [6, 21, 55, 52], sort the order of task learning [16], or adapt the feature sharing between ‘related’ and ‘unrelated’ tasks [65], to mitigate the effect of negative transfer.”
Any-Precision Deep Neural Networks (2019, methodology): “Inspired by (Zhao et al. 2018), we analyze gradient consistency between bit-widths by computing Update Compliance Average (UCA), which is defined as the average of cosine similarity of gradients from two bit-widths over multiple training steps.”
Multi-Task Learning for Dense Prediction Tasks: A Survey (2020, background): “To this end, other works [22], [26], [69] have also considered the influence of the direction of the task gradients.”
Branched Multi-Task Networks: Deciding what layers to share (2019, background): “This situation hurts performance, as a suboptimal grouping of tasks can lead to the sharing of information between unrelated tasks, known as negative transfer [47].”
Towards Adversarially Robust Object Detection (2019, background): “Furthermore, by viewing object detection as an instance of multi-task learning task, this work could serve as an example on robustness improvement for other multi-task learning problems as well [20, 59].”
Multi-task Learning (2020, methodology): “Another set of methods seeks to mitigate the optimization challenges in multi-task learning by manipulating the task gradients in a number of ways such as (1) modifying the direction of task gradients with the underlying assumption that directional inconsistency of gradients on the shared parameters”
Transferability and Hardness of Supervised Classification Tasks (2019, background): “Finally, in some scenarios, establishing task relationships requires multi-task learning of the models, to measure the influence different tasks have on each other [30, 36, 59, 66, 71].”
Contextualizing Multiple Tasks via Learning to Decompose (2021, background): “To explore the rich semantics [34, 66], non-euclidean relationship [5, 2] and conditional similarity [32] are investigated.”
Attentive Single-Tasking of Multiple Tasks (2019, background): “Several works study modulation for image retrieval [69] or classiﬁ-cation tasks [44, 40], and embeddings for different artistic styles [11].”
Efficiently Identifying Task Groupings for Multi-Task Learning (2021, unknown): “Another set of methods seek to mitigate inter-task conﬂict by manipulating the direction of task gradients rather than simply their magnitude [48, 57, 53, 11, 50, 36].”
Multi-Task Learning with Deep Neural Networks: A Survey (2020, background): “(Zhang et al., 2014; Dai et al., 2016; Zhao et al., 2018; Liu et al., 2019; Ma et al., 2018) propose architectures which are variations on the shared trunk idea.”