Integrated perception with recurrent multi-task neural networks

By Hakan Bilen, Andrea Vedaldi. 2016

98 citations recorded by the source corpus.

Modern discriminative predictors have been shown to match natural intelligences in specific perceptual tasks in image classification, object and part detection, boundary extraction, etc. However, a major advantage that natural intelligences still have is that they work well for "all" perceptual problems together, solving them efficiently and coherently in an "integrated manner". In order to capture some of these advantages in machine perception, we ask two questions: whether deep neural networks can learn universal image representations, useful not only for a single task but for all of them, and how the solutions to the different tasks can be integrated in this framework. We answer by proposing a new architecture, which we call "MultiNet", in which not only deep image features are shared between tasks, but where tasks can interact in a recurrent manner by encoding the results of their analysis in a common shared representation of the data. In this manner, we show that the performance of individual tasks in standard benchmarks can be improved first by sharing features between them and then, more significantly, by integrating their solutions in the common representation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

UberNet: Training a Universal Convolutional Neural Network for Low-, Mid-, and High-Level Vision Using Diverse Datasets and Limited Memory (2016, background): “…and region proposal generation; [70] study the effects of sharing information across networks trained for complementary tasks, while more recently [8] propose the introduction of inter-task connections that can improve performance through task synergy, while [84] propose an architecture…”
Incremental Learning Through Deep Adaptation (2017, background): “Some examples include facial landmark localization [7], semantic segmentation [8], 3D-reasoning [9], object and part detection [10] and others.”
Efficient Parametrization of Multi-domain Deep Neural Networks (2018, methodology): “In computer vision, deep MTL models are applied to object tracking [34], facial-landmark detection [35], object and part detection [2], object detection and instance segmentation [10], a collection of low-level and high-level vision tasks [14].”
Gradient Surgery for Multi-Task Learning (2020, methodology): “…Zamir et al., 2018) and reinforcement-learning (Espeholt et al., 2018; Wilson et al., 2007), as well as many different domains, such as vision (Bilen & Vedaldi, 2016; Misra et al., 2016a; Kokkinos, 2017; Liu et al., 2018; Zamir et al., 2018), language (Col-lobert & Weston, 2008; Dong et al.,…”
Taskonomy: Disentangling Task Transfer Learning (2018, methodology): “[9 3]”
Playing for Benchmarks (2017, background): “Researchers have argued that diverse computer vision tasks should be tackled in concert [39] and developed models that could support this [6, 33, 43].”
Which Tasks Should Be Learned Together in Multi-task Learning? (2019, methodology): “Hard parameter sharing is also used in many other works such as [42, 4, 27, 12, 17, 2, 33, 10, 45, 19, 23, 8, 47, 36].”
Beyond Shared Hierarchies: Deep Multitask Learning through Soft Layer Ordering (2017, methodology): “Such techniques have been applied across settings such as vision (Bilen and Vedaldi, 2016; 2017; Jou and Chang, 2016; Lu et al., 2017; Misra et al., 2016; Ranjan et al., 2016; Yang and Hospedales, 2017; Zhang et al., 2014), natural language (Collobert and Weston, 2008; Dong et al., 2015; Hashimoto…”
Fully-Adaptive Feature Sharing in Multi-Task Networks with Applications in Person Attribute Classification (2016, background): “MultiNet [3] exploits recurrent networks for transferring information across tasks.”
Multi-Task Learning as Multi-Objective Optimization (2018, methodology): “We focus on hard parameter sharing with gradient-based optimization, following the success of deep MTL in computer vision (Bilen and Vedaldi, 2016; Misra et al., 2016; Rudd et al., 2016; Yang and Hospedales, 2016; Kokkinos, 2017; Zamir et al., 2018), natural language processing (Collobert and…”
Visual Reasoning by Progressive Module Networks (2018, background): “Bilen & Vedaldi (2016) propose the Multinet architecture where tasks can interact with each other in addition to shared image features.”
Dynamic Task Prioritization for Multitask Learning (2018, background): “Hard parameter sharing has been successful in image classification [54], object detection [55, 39], semantic segmentation [53], and facial analysis [56].”
Stochastic Filter Groups for Multi-Task CNNs: Learning Specialist and Generalist Convolution Kernels (2019, background): “Recently, applications of convolutional neural networks (CNNs) in MTL have demonstrated promising results in a wide-range of computer vision applications, ranging from visual scene understanding [2, 3, 4, 5, 6, 7] to medical image computing [8, 9, 10, 11].”
AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning (2019, background): “Hard-parameter sharing is the most common MTL approach that usually relies on hand-designed network architectures composed of hidden layers that are shared across all tasks and specialized branches that learn task-speciﬁc features [19, 26, 39, 5, 22, 11].”
Piggyback: Adding Multiple Tasks to a Single, Fixed Network by Learning to Mask (2018, background): “While multiple prior works [1, 2, 16] have explored multi-task training, wherein data of all tasks is available at the time of training, we consider the setting in which new tasks are available sequentially, as this is a more realistic and challenging scenario.”
Universal representations: The missing link between faces, text, planktons, and cat breeds (2017, methodology): “Multi-task learning is applied to various computer vision problems and reported to achieve performance gains in object tracking [60], faciallandmark detection [61], surface normals and edge labels [57], object detection and segmentation [10] object and part detection [3].”
Piggyback: Adapting a Single Network to Multiple Tasks by Learning to Mask Weights (2018, background): “While multiple prior works [18,19,20] have explored multi-task training, wherein data of all tasks is available at the time of training, we consider the setting in which new tasks are available sequentially, a more realistic and challenging scenario.”