Is Multi-Task Learning an Upper Bound for Continual Learning?

By Zihao Wu, Huy Tran, Hamed Pirsiavash, Soheil Kolouri. 2022

7 citations recorded by the source corpus.

Continual learning and multi-task learning are commonly used machine learning techniques for learning from multiple tasks. However, existing literature assumes multi-task learning as a reasonable performance upper bound for various continual learning algorithms, without rigorous justification. Additionally, in a multi-task setting, a small subset of tasks may behave as adversarial tasks, negatively impacting overall learning performance. On the other hand, continual learning approaches can avoid the negative impact of adversarial tasks and maintain performance on the remaining tasks, resulting in better performance than multi-task learning. This paper introduces a novel continual self-supervised learning approach, where each task involves learning an invariant representation for a specific class of data augmentations. We demonstrate that this approach results in naturally contradicting tasks and that, in this setting, continual learning often outperforms multi-task learning on benchmark datasets, including MNIST, CIFAR-10, and CIFAR-100.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

  1. Liangzu Peng
  2. Paris V. Giampouras
  3. René Vidal
  4. Ehsan Elhamifar
  5. Paolo Favaro
  6. Guangcan Liu
  7. Zhouchen Lin
  8. M. Soltanolkotabi

Citations in the corpus