MultiNet: Real-time Joint Semantic Reasoning for Autonomous Driving

By Marvin T. T. Teichmann, Michael Weber, Johann Marius Zöllner, Roberto Cipolla, Raquel Urtasun. 2016

772 citations recorded by the source corpus.

While most approaches to semantic reasoning have focused on improving performance, in this paper we argue that computational times are very important in order to enable real time applications such as autonomous driving. Towards this goal, we present an approach to joint classification, detection and semantic segmentation using a unified architecture where the encoder is shared amongst the three tasks. Our approach is very simple, can be trained end-to-end and performs extremely well in the challenging KITTI dataset. Our approach is also very efficient, allowing us to perform inference at more then 23 frames per second. Training scripts and trained weights to reproduce our results can be found here: https://github.com/MarvinTeichmann/MultiNet

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Unified Perceptual Parsing for Scene Understanding (2018, methodology): “[24] propose an approach to perform classification, detection, and semantic segmentation via a shared feature extractor.”
WoodScape: A Multi-Task, Multi-Camera Fisheye Dataset for Autonomous Driving (2019, background): “However, there is a recent trend [30, 53, 51, 8] to solve tasks using a single multi-task model to enable efficient reuse of encoder features and also”
Multi-task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics (2017, methodology): “This is the dominant approach used by prior work [39, 38, 30, 41], for example for dense prediction tasks [27], for scene understanding tasks [15] and for rotation (in quaternions) and translation (in meters) for camera pose [25].”
GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks (2017, methodology): “Deep networks have been used to solve various subsets of multiple vision tasks, from 3-task networks (Eigen & Fergus (2015); Teichmann et al. (2016)) to much larger subsets as in UberNet (Kokkinos (2016)).”
Learning to Fuse Things and Stuff (2018, background): “As the corresponding architectures share fundamental similarities inherited from the seminal AlexNet model [17], several works have naturally leveraged the commonalities to propose multi-task models that can simultaneously address semantic segmentation, object detection, and more [39, 14, 16, 28, 35”
Certifiably Robust Interpretation in Deep Learning (2019, background): “The growing use of deep learning in many sensitive areas like autonomous driving, medicine, finance and even the legal system ([1, 2, 3, 4]) raises concerns about human trust in machine learning systems.”
LIDAR-Camera Fusion for Road Detection Using Fully Convolutional Neural Networks (2018, neutral): “Teichmann et al. [7], for example, trained a convolutional neural network (CNN) to simultaneously solve the tasks of road segmentation and vehicle detection in monocular camera images.”
Deep Multi-Modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges (2019, background): “For example, [54], [60], [61] focus on pixel-wise semantic segmentation for multiple classes including road, car, bicycle, column-pole, tree, sky, etc; [52] and [62] concentrate on road segmentation; and [51], [63], [64] deal”
Stacked Conditional Generative Adversarial Networks for Jointly Learning Shadow Detection and Shadow Removal (2017, methodology): “For example, in Mask R-CNN [13] and MultiNet [48], 3 parallel branches for object classification, bounding-box regression and semantic segmentation respectively are utilized.”
What Should Not Be Contrastive in Contrastive Learning (2020, methodology): “Multi-task learning has been widely applied in image recognition (Kokkinos, 2017; Teichmann et al., 2018; He et al., 2017).”
NeurAll: Towards a Unified Model for Visual Perception in Automated Driving (2019, background): “Uniﬁed models learn features jointly for all tasks which makes them robust to over-ﬁtting by acting as a regularizer, as demonstrated in various multi-task networks [27], [3], [28].”
Multi-Scale Progressive Fusion Network for Single Image Deraining (2020, background): “Due to substantial degradation of the image content in rain images and videos, traditional image enhancement algorithms [27] struggle to make desirable improvements on image quality.”
Deep semantic segmentation for automated driving: Taxonomy, roadmap and challenges (2017, methodology): “Another potential solution is to use mask predictions on detected bounding boxes of these small objects as in [50][51].”
Cross-View Semantic Segmentation for Sensing Surroundings (2019, background): “rks enables machine to segment a scene precisely into meaningful regions and objects [46, 24]. So-called semantic segmentation networks have been widely used in many ﬁelds, such as autonomous driving [12, 35], human-computer interaction and robotics [38, 37, 34]. Though the semantic segmentation net”
Generative Adversarial Networks (GANs) (2020, background): “Apart from it, GANs has also shown its potential for several other applications, such as Steganalysis [150], information retrieval [151], spatio-temporal data prediction [152], autonomous driving [153]–[155], speech enhancement [156], etc.”
BlitzNet: A Real-Time Deep Network for Scene Understanding (2017, background): “Closely related to our work, but dedicated to autonomous driving, [28] also proposes to integrate semantic segmentation and object detection via a deep network.”
NeurAll: Towards a Unified Visual Perception Model for Automated Driving (2019, background): “[28] and fast scene understanding by Neven et al.”