Deep Imitative Models for Flexible Inference, Planning, and Control

By Nicholas Rhinehart, Rowan Thomas McAllister, Sergey Levine. 2018

160 citations recorded by the source corpus.

Imitation Learning (IL) is an appealing approach to learn desirable autonomous behavior. However, directing IL to achieve arbitrary goals is difficult. In contrast, planning-based algorithms use dynamics models and reward functions to achieve goals. Yet, reward functions that evoke desirable behavior are often difficult to specify. In this paper, we propose Imitative Models to combine the benefits of IL and goal-directed planning. Imitative Models are probabilistic predictive models of desirable behavior able to plan interpretable expert-like trajectories to achieve specified goals. We derive families of flexible goal objectives, including constrained goal regions, unconstrained goal sets, and energy-based goals. We show that our method can use these objectives to successfully direct behavior. Our method substantially outperforms six IL approaches and a planning-based approach in a dynamic simulated autonomous driving task, and is efficiently learned from expert demonstrations without online data collection. We also show our approach is robust to poorly specified goals, such as goals on the wrong side of the road.

Subfields: Reinforcement Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

In Meta & Continual Learning

Citations in the corpus

Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, background): “…close to the training data for the model, so as to avoid out-of-distribution inputs as discussed in Section 5, can effectively provide elements of imitation learning when training on driving demonstration data, as for example in the case of deep imitative models (DIMs) (Rhinehart et al., 2018).”
Continual Learning for Robotics (2019, background): “This as well happens for autonomous robots that learn new manipulation tasks, for exploration policies, as well as for autonomous vehicles that need to learn and adapt to new circumstances (Bojarski et al., 2016; Codevilla et al., 2017; Jaritz et al., 2018; Rhinehart et al., 2018).”
Improvisation through Physical Understanding: Using Novel Objects as Tools with Visual Foresight (2019, methodology): “Learning from demonstrations has also been used in combination with planning , where a planning cost function is inferred from data [38, 47, 1, 26] or where tool-use is learned”
TransFuser: Imitation With Transformer-Based Sensor Fusion for Autonomous Driving (2022, background): “We consider the task of point-to-point navigation in an urban setting [1], [2], [38], [86], [87] where the goal is to complete a given route while safely reacting to other dynamic agents and following trafﬁc rules.”
PRECOG: PREdiction Conditioned on Goals in Visual Multi-Agent Settings (2019, methodology): “While [31] could implicitly model other agents through its visual conditioning, we show explicit modeling of other agents yields better forecasting results, in addition to giving us the tools to predict responses to agent’s plans.”
Recent Advancements in End-to-End Autonomous Driving Using Deep Learning: A Survey (2023, background): “TO R EAL Large-scale virtual scenarios can be constructed in virtual engines, enabling the collection of a signiﬁcant quantity of data more readily.”
A Survey of State-Action Representations for Autonomous Driving (2018, unknown): “This representation is used by (Ziebart et al., 2009; Richter et al., 2014; Chen et al., 2016; Tamar et al., 2016; Shankar et al., 2016; Wulfmeier et al., 2016; Sallab et al., 2017; Rehder et al., 2017a; Hoermann et al., 2017; Williams et al., 2017; Mukadam et al., 2018; Rhinehart et al., 2018).”
Parrot: Data-Driven Behavioral Priors for Reinforcement Learning (2020, methodology): “Other works learn these distributions from data (Xie et al., 2019; Rhinehart et al., 2020) and utilize them for planning at test time to optimize a user-provided cost function.”
Human motion trajectory prediction: a survey (2019, methodology): “A semantic map (Karasev et al. 2016; Kitani et al. 2012; Rehder et al. 2018; Coscia et al. 2018; Shen et al. 2018; Vasishta et al. 2017, 2018; Rhinehart et al. 2018; Rhinehart et al. 2018; Tadokoro et al. 1993; Ballan et al. 2016; Zhao et al. 2019) or extracted features from an image (Xue et al.…”
Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baseline (2022, unknown): “The output prediction of the model for end-to-end autonomous driving generally falls into two forms: trajectory/waypoints [48, 4, 11, 46, 15, 29, 10] and direct control actions [17, 39, 18, 42, 12, 60, 9].”
MP3: A Unified Model to Map, Perceive, Predict and Plan (2021, background): “Following [42], we assume a separate camera-based perception system detects the trafﬁc lights and recognizes their state as this is not our focus.”
ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst (2018, unknown): “[20]), and motion planning (Paden et al.”
INTERACTION Dataset: An INTERnational, Adversarial and Cooperative moTION Dataset in Interactive Driving Scenarios with Semantic Maps (2019, background): “…road users [1]–[3], ii) modeling and analyzing driving behavior [4], [5], iii) clustering the motion and ﬁnding representation of the motion primitives [6], [7], iv) cloning and imitating human and expert behavior [8], [9], and v) generating human-like and social behavior and motion [10]–[12].”
Multi-Modal Fusion Transformer for End-to-End Autonomous Driving (2021, background): “We consider the task of point-to-point navigation in an urban setting [23, 45, 46, 8, 16] where the goal is to complete a given route while safely reacting to other dynamic agents and following traffic rules.”
Conditional Vehicle Trajectories Prediction in CARLA Urban Environment (2019, methodology): “[17, 28] use Lidar information in combination with image, either with imitation or model based reinforcement learning.”
Core Challenges of Social Robot Navigation: A Survey (2021, background): “Note that recent research in neighboring fields such as autonomous driving and human motion prediction is also dominated by approaches that strive to leverage models of coupled interaction [31, 152, 154, 167, 207, 210, 239].”