NDDR-CNN: Layerwise Feature Fusing in Multi-Task CNNs by Neural Discriminative Dimensionality Reduction

By Yuan Gao, Qi She, Jiayi Ma, Mingbo Zhao, Wei David Liu, Alan L. Yuille. Computer Vision and Pattern Recognition, 2018

306 citations recorded by the source corpus.

In this paper, we propose a novel Convolutional Neural Network (CNN) structure for general-purpose multi-task learning (MTL), which enables automatic feature fusing at every layer from different tasks. This is in contrast with the most widely used MTL CNN structures which empirically or heuristically share features on some specific layers (e.g., share all the features except the last convolutional layer). The proposed layerwise feature fusing scheme is formulated by combining existing CNN components in a novel way, with clear mathematical interpretability as discriminative dimensionality reduction, which is referred to as Neural Discriminative Dimensionality Reduction (NDDR). Specifically, we first concatenate features with the same spatial resolution from different tasks according to their channel dimension. Then, we show that the discriminative dimensionality reduction can be fulfilled by 1x1 Convolution, Batch Normalization, and Weight Decay in one CNN. The use of existing CNN components ensures the end-to-end training and the extensibility of the proposed NDDR layer to various state-of-the-art CNN architectures in a "plug-and-play" manner. The detailed ablation analysis shows that the proposed NDDR layer is easy to train and also robust to different hyperparameters. Experiments on different task sets with various base network architectures demonstrate the promising performance and desirable generalizability of our proposed method. The code of our paper is available at https://github.com/ethanygao/NDDR-CNN.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/8c8d0031b24937d8a8ec7a4c5ab5fda4f4797803

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning (2019, methodology): “Furthermore, our model has a signiﬁcantly lower number of parameters (about 50% lower while learning two tasks) compared to the recently proposed deep multi-task learning methods [15, 31].”
Training-Time-Friendly Network for Real-Time Object Detection (2019, background): “It is an elegant and efﬁcient solution for balancing training time, inference speed, and accuracy, which can provide more potentials and possibilities for training-time-sensitive tasks(Zoph and Le 2017; Zoph et al. 2018; Ghiasi, Lin, and Le 2019; Wang et al. 2019b; Gao et al. 2019).”
Branched Multi-Task Networks: Deciding what layers to share (2019, result): “We also compare our branched multi-task networks with cross-stitch networks [33], NDDR-CNNs [13] and MTAN [26] in Table 2.”
MTL-NAS: Task-Agnostic Neural Architecture Search Towards General-Purpose Multi-Task Learning (2020, methodology): “ulti-task architecture into a (general) shared structure involving feature fusion connections and the task-speciﬁc sub-networks, and optimize the shared structure with NAS. Recent researches in GPMTL [13,42] inspire us to formulate the single-task backbones as the task-speciﬁc parts, while focusing ”
Multi-Task Learning for Dense Prediction Tasks: A Survey (2020, methodology): “NDDR-CNN [7] also incorporated dimensionality reduction techniques into the feature fusion”