CIRL: Controllable Imitative Reinforcement Learning for Vision-based Self-driving

By Xiaodan Liang, Tairui Wang, Luona Yang, Eric P. Xing. 2018

310 citations recorded by the source corpus.

Autonomous urban driving navigation with complex multi-agent dynamics is under-explored due to the difficulty of learning an optimal driving policy. The traditional modular pipeline heavily relies on hand-designed rules and the pre-processing perception system while the supervised learning-based models are limited by the accessibility of extensive human experience. We present a general and principled Controllable Imitative Reinforcement Learning (CIRL) approach which successfully makes the driving agent achieve higher success rates based on only vision inputs in a high-fidelity car simulator. To alleviate the low exploration efficiency for large continuous action space that often prohibits the use of classical RL on challenging real tasks, our CIRL explores over a reasonably constrained action space guided by encoded experiences that imitate human demonstrations, building upon Deep Deterministic Policy Gradient (DDPG). Moreover, we propose to specialize adaptive policies and steering-angle reward designs for different control signals (i.e. follow, straight, turn right, turn left) based on the shared representations to improve the model capability in tackling with diverse cases. Extensive experiments on CARLA driving benchmark demonstrate that CIRL substantially outperforms all previous methods in terms of the percentage of successfully completed episodes on a variety of goal-directed driving tasks. We also show its superior generalization capability in unseen environments. To our knowledge, this is the first successful case of the learned driving policy by reinforcement learning in the high-fidelity simulator, which performs better than supervised imitation learning.

Subfields: Artificial Intelligence, Automotive Engineering, Social Psychology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Automotive Engineering

In Social Psychology

Citations in the corpus

Deep Imitative Models for Flexible Inference, Planning, and Control (2018, methodology): “Therefore, MBRL algorithms usually require online collection and training (Englert et al., 2013; Liang et al., 2018).”
Feature Selective Anchor-Free Module for Single-Shot Object Detection (2019, background): “It serves as a prerequisite for various downstream vision applications such as instance segmentation [12], facial analysis [1, 39], autonomous driving cars [6, 20], and video analysis [25, 33].”
Recent Advancements in End-to-End Autonomous Driving Using Deep Learning: A Survey (2023, background): “TO R EAL Large-scale virtual scenarios can be constructed in virtual engines, enabling the collection of a signiﬁcant quantity of data more readily.”
End-to-End Urban Driving by Imitating a Reinforcement Learning Coach (2021, background): “CIRL [27] proposed an end-to-end DDPG [28] agent with its actor network pre-trained via behavior cloning to accelerate online training.”
End-to-End Autonomous Driving: Challenges and Frontiers (2023, background): “RL can also be used to ﬁnetune full networks that were pre-trained using IL [17], [79].”
Learning by Cheating (2019, background): “Task Weather MP[8] CIL[6] CIRL[13] CAL[20] CILRS[7] LBC LBC† Straight train 92 97 100 93 96 100 100 One turn 61 59 71 82 84 100 100 Navigation 24 40 53 70 69 100 98 Nav.”
Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baseline (2022, background): “The output prediction of the model for end-to-end autonomous driving generally falls into two forms: trajectory/waypoints [48, 4, 11, 46, 15, 29, 10] and direct control actions [17, 39, 18, 42, 12, 60, 9].”
Learning to Rank Proposals for Object Detection (2019, background): “Object detection is a fundamental yet challenging task in computer vision, It is extensively applied in video/image indexing [29] [24], face recognition [31], autonomous driving cars [7] [19] and human pose estimation [32].”
Planning-oriented Autonomous Driving (2022, background): “Subsequent studies make great advances especially in closed-loop simulation with deeper networks [4], multi-modal inputs [2, 21, 78], multi-task learning [20,97], reinforcement learning [13,14, 46, 59, 89] and distillation from certain privilege knowledge [16,103,106].”
ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst (2018, background): “[13]), hybrids of reinforcement and imitation learning (Kuefler et al.”
Exploring the Limitations of Behavior Cloning for Autonomous Driving (2019, methodology): “New weather New town Task CIL [10] CIRL [26] CAL [36] MT [25] CILR CILRS CIL [10] CIRL [26] CAL [36] MT [25] CILR CILRS”
Reward (Mis)design for Autonomous Driving (2021, background): “CIRL: controllable imitative reinforcement learning for vision-based self-driving [29]”
Multimodal End-to-End Autonomous Driving (2019, methodology): “In order to answer the posed question, as is being common practice in many recent previous works on end-to-end driving [71], [68], [67], [69], [72], [73], [80], we will rely on the open source driving simulator known as CARLA [98] and its standard benchmark.”
A Survey of End-to-End Driving: Architectures and Training Methods (2020, methodology): “in [9]), one might also just explicitly punish crashes by assigning a large negative reward, as was done in [26], [61], [66], [69], [70].”