Multi-Task Learning as Multi-Objective Optimization

By Ozan Sener, Vladlen Koltun. 2018

1,788 citations recorded by the source corpus.

In multi-task learning, multiple tasks are solved jointly, sharing inductive bias between them. Multi-task learning is inherently a multi-objective problem because different tasks may conflict, necessitating a trade-off. A common compromise is to optimize a proxy objective that minimizes a weighted linear combination of per-task losses. However, this workaround is only valid when the tasks do not compete, which is rarely the case. In this paper, we explicitly cast multi-task learning as multi-objective optimization, with the overall objective of finding a Pareto optimal solution. To this end, we use algorithms developed in the gradient-based multi-objective optimization literature. These algorithms are not directly applicable to large-scale learning problems since they scale poorly with the dimensionality of the gradients and the number of tasks. We therefore propose an upper bound for the multi-objective loss and show that it can be optimized efficiently. We further prove that optimizing this upper bound yields a Pareto optimal solution under realistic assumptions. We apply our method to a variety of multi-task deep learning problems including digit classification, scene understanding (joint semantic segmentation, instance segmentation, and depth estimation), and multi-label classification. Our method produces higher-performing models than recent multi-task learning formulations or per-task training.

Subfields: Artificial Intelligence, Computational Mechanics, Computer Vision and Pattern Recognition, Optimization & Training, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Sergey Levine
  2. Yoshua Bengio
  3. Quoc V. Le
  4. Mohit Bansal
  5. Jianfeng Gao
  6. Dacheng Tao
  7. Pieter Abbeel
  8. Chelsea Finn

In Computational Mechanics

  1. Luc Van Gool
  2. Qiang Yang
  3. Iasonas Kokkinos
  4. Kevis-Kokitsi Maninis
  5. Simon Vandenhende
  6. Michael Crawshaw
  7. Stamatios Georgoulis
  8. Yu Zhang

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Trevor Darrell
  4. Xiaogang Wang
  5. Jitendra Malik
  6. Ming-Hsuan Yang
  7. Yi Yang
  8. Andrew Zisserman

In Optimization & Training

  1. Michael Crawshaw
  2. Sergey Levine
  3. Chelsea Finn
  4. Vladlen Koltun
  5. Tianbao Yang
  6. Zhe Zhao
  7. Tianhe Yu
  8. Abhishek Gupta

In Meta & Continual Learning

  1. Chelsea Finn
  2. Sergey Levine
  3. Tianhe Yu
  4. Michael Crawshaw
  5. Abhishek Gupta
  6. Luc Van Gool
  7. Maliheh Izadi
  8. Roberta Gismondi

Citations in the corpus