MultiNet: Multi-Modal Multi-Task Learning for Autonomous Driving

By Sauhaarda Chowdhuri, Tushar Pankaj, Karl Zipser. 2017

81 citations recorded by the source corpus.

Autonomous driving requires operation in different behavioral modes ranging from lane following and intersection crossing to turning and stopping. However, most existing deep learning approaches to autonomous driving do not consider the behavioral mode in the training strategy. This paper describes a technique for learning multiple distinct behavioral modes in a single deep neural network through the use of multi-modal multi-task learning. We study the effectiveness of this approach, denoted MultiNet, using self-driving model cars for driving in unstructured environments such as sidewalks and unpaved roads. Using labeled data from over one hundred hours of driving our fleet of 1/10th scale model cars, we trained different neural networks to predict the steering angle and driving speed of the vehicle in different behavioral modes. We show that in each case, MultiNet networks outperform networks trained on individual modes while using a fraction of the total number of parameters.

Subfields: Automotive Engineering, Computer Vision and Pattern Recognition, Safety, Risk, Reliability and Quality

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Automotive Engineering

1. Kashyap Chitta
2. Andreas Geiger
3. Hongyang Li
4. Li Chen
5. Jacob Lambert
6. Alexander Carballo
7. Peng Wu
8. Ekim Yurtsever

In Computer Vision and Pattern Recognition

1. Hongyang Li
2. Kashyap Chitta
3. Ekim Yurtsever
4. Jacob Lambert
5. Alexander Carballo
6. Li Chen
7. Peng Wu
8. Kazuya Takeda

In Safety, Risk, Reliability and Quality

1. Kazuya Takeda
2. Alexander Carballo
3. Ekim Yurtsever
4. Kashyap Chitta
5. Andreas Geiger
6. Jacob Lambert

Citations in the corpus

• A Survey of Autonomous Driving: Common Practices and Emerging Technologies (2019, background): “object dynamics are also emerging [196] which further suggests that generalized networks tackling multiple perception tasks may be the future of ADS perception.”
• A Survey of End-to-End Driving: Architectures and Training Methods (2020, methodology): “ordered to keep to the right of the lane, to the center of the lane or follow another vehicle [48] via additional categorical inputs.”
• Domain-specific Genetic Algorithm for Multi-tenant DNNAccelerator Scheduling (2021, background): “First, end applications (such as AR/VR [55] or self-driving [16, 117]) and cloud services (such as search [36, 124] and translation [58, 62]) often rely on several DNN models internally.”
• End-to-End Autonomous Driving: Challenges and Frontiers (2023, background): “Naive concatenation is also frequently adopted [15], [22], [30], [142], [143], [144], [145], [146].”