UberNet: Training a Universal Convolutional Neural Network for Low-, Mid-, and High-Level Vision Using Diverse Datasets and Limited Memory

By Iasonas Kokkinos. 2016

730 citations recorded by the source corpus.

In this work we train in an end-to-end manner a convolutional neural network (CNN) that jointly handles low-, mid-, and high-level vision tasks in a unified architecture. Such a network can act like a swiss knife for vision tasks, we call it an UberNet to indicate its overarching nature. The main contribution of this work consists in handling challenges that emerge when scaling up to many tasks. We introduce techniques that facilitate (i) training a deep architecture while relying on diverse training sets and (ii) training many (potentially unlimited) tasks with a limited memory budget. This allows us to train in an end-to-end manner a unified CNN architecture that jointly handles (a) boundary detection (b) normal estimation (c) saliency estimation (d) semantic segmentation (e) human part segmentation (f) semantic boundary detection, (g) region proposal generation and object detection. We obtain competitive performance while jointly addressing all tasks in 0.7 seconds on a GPU. Our system will be made publicly available.

Subfields: Computer Vision and Pattern Recognition, Industrial and Manufacturing Engineering, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Industrial and Manufacturing Engineering

In Model Architectures

Citations in the corpus

Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning (2018, background): “Our work is most similar to that of Luong et al. (2015), who train a many-to-many sequenceto-sequence model on a diverse set of weakly related tasks that includes machine translation, constituency parsing, image captioning, sequence autoencoding, and intra-sentence skip-thoughts. There are two key d”
Unified Perceptual Parsing for Scene Understanding (2018, background): “Moreover, although tasks in [25] are formed from low level to high level, such as boundary detection, semantic segmentation and object detection, these tasks do not form the hierarchy of visual concepts.”
Pixelwise Instance Segmentation with a Dynamically Instantiated Network (2017, methodology): “Possible techniques for doing this are suggested by [25] and [39].”
Perceiver IO: A General Architecture for Structured Inputs & Outputs (2021, background): “In vision, multi-task learning has become popular [22, 40, 56, 100].”
Multi-task Self-Supervised Visual Learning (2017, background): “Kokkinos [18], for example, found a slight dip in performance with 7 tasks versus 2.”
WoodScape: A Multi-Task, Multi-Camera Fisheye Dataset for Autonomous Driving (2019, background): “However, there is a recent trend [30, 53, 51, 8] to solve tasks using a single multi-task model to enable efficient reuse of encoder features and also”
12-in-1: Multi-Task Vision and Language Representation Learning (2019, background): “Advances in multi-task learning have been developed in the context of vision [5,20,32,42,52,53], language [10, 25, 26, 31, 37], and robotics [18, 34, 46].”
Reparameterizing Convolutions for Incremental Multi-Task Learning without Task Interference (2020, background): “roduction. When it comes to learning multiple tasks under a single model, multi-task learning (MTL) techniques [2,54] have been employed in the literature. On the one hand, encoder-focused approaches [41,25,36,10,43,33,1,61] emphasize learning feature representations from multi-task supervisory sign”
Expert Gate: Lifelong Learning with a Network of Experts (2016, background): “Unlike [16], who train one Uber network for performing vision tasks as diverse as semantic segmentation, object detection and human body part detection, our work focuses on tasks with a similar objective.”
2D/3D Pose Estimation and Action Recognition Using Multitask Deep Learning (2018, background): “As suggested by Kokkinos [24], this is all the more true for multitask problems, where related tasks can beneﬁt from one another.”
Gradient Surgery for Multi-Task Learning (2020, methodology): “The multi-task formulation has been applied to many different settings, including supervised learning [63, 35, 60, 53, 62] and reinforcement-learning [17, 58], as well as many different domains, such as vision [3, 39, 31, 33, 62], language [11, 15, 38, 44] and robotics [45, 59, 25].”
Panoptic-DeepLab: A Simple, Strong, and Fast Baseline for Bottom-Up Panoptic Segmentation (2019, methodology): “Finally, our model employs the parallel multi-head prediction framework similar to [68, 36, 55].”