Reparameterizing Convolutions for Incremental Multi-Task Learning without Task Interference

By Menelaos Kanakis, David Bruggemann, Suman Saha, Stamatios Georgoulis, Anton Obukhov, Luc Van Gool. 2020

83 citations recorded by the source corpus.

Multi-task networks are commonly utilized to alleviate the need for a large number of highly specialized single-task networks. However, two common challenges in developing multi-task models are often overlooked in literature. First, enabling the model to be inherently incremental, continuously incorporating information from new tasks without forgetting the previously learned ones (incremental learning). Second, eliminating adverse interactions amongst tasks, which has been shown to significantly degrade the single-task performance in a multi-task setup (task interference). In this paper, we show that both can be achieved simply by reparameterizing the convolutions of standard neural network architectures into a non-trainable shared part (filter bank) and task-specific parts (modulators), where each modulator has a fraction of the filter bank parameters. Thus, our reparameterization enables the model to learn new tasks without adversely affecting the performance of existing ones. The results of our ablation study attest the efficacy of the proposed reparameterization. Moreover, our method achieves state-of-the-art on two challenging multi-task learning benchmarks, PASCAL-Context and NYUD, and also demonstrates superior incremental learning capability as compared to its close competitors.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “Besides, the network parameters themselves can be decomposed into task-sharing and task-specific elements, such as by additive decomposition (APD [479]), singular value decomposition (RCM [197]), filter atom decomposition (FAS [300]) and lowrank factorization (IBP-WF [294], IRU [177]).”
Representation Compensation Networks for Continual Semantic Segmentation (2022, methodology): “Some other methods [45,57,75,76,82,91] increase the capacity of the model to have a better tradeoff of stability and plasticity, but with the cost of growing memory of the network.”
Recent Advances of Continual Learning in Computer Vision: An Overview (2021, positive): “[149] proposed Reparameterized Convolutions for Multitask learning (RCM) which separates the model parameters into filter-bank part and modulator part.”
MgSvF: Multi-Grained Slow versus Fast Framework for Few-Shot Class-Incremental Learning (2020, background): “Incremental learning is usually conducted under the task-incremental [72], [73] or the class-incre-mental learning scenarios [2], [70], [74], [75], [76], [77].”
Factorizing Knowledge in Neural Networks (2022, background): “Some recent solutions explore on the decomposition between shared and task-speciﬁc processing [40,25,69].”
Endpoints Weight Fusion for Class Incremental Semantic Segmentation (2023, methodology): “Some methods [27, 33, 42, 47] propose to expand the model in incremental steps and ensemble the old and new outputs, which have large memory and inference costs.”
Continual Learning with Foundation Models: An Empirical Study of Latent Replay (2022, background): “This effect is known as interference (Kanakis et al., 2020; Kokkinos, 2017).”