Imitation learning for locomotion and manipulation

By Nathan D. Ratliff, J. Bagnell, Siddhartha S. Srinivasa. 2007

139 citations recorded by the source corpus.

Decision making in robotics often involves computing an optimal action for a given state, where the space of actions under consideration can potentially be large and state dependent. Many of these decision making problems can be naturally formalized in the multiclass classification framework, where actions are regarded as labels for states. One powerful approach to multiclass classification relies on learning a function that scores each action; action selection is done by returning the action with maximum score. In this work, we focus on two imitation learning problems in particular that arise in robotics. The first problem is footstep prediction for quadruped locomotion, in which the system predicts next footstep locations greedily given the current four-foot configuration of the robot over a terrain height map. The second problem is grasp prediction, in which the system must predict good grasps of complex free-form objects given an approach direction for a robotic hand. We present experimental results of applying a recently developed functional gradient technique for optimizing a structured margin formulation of the corresponding large non-linear multiclass classification problems.

Subfields: Artificial Intelligence, Control and Systems Engineering, Reinforcement Learning, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Reinforcement Learning

In Robotics & Control

Citations in the corpus

Reinforcement learning in robotics: A survey (2013, methodology): “These techniques, and many variants, have been recently successfully applied to outdoor robot navigation (Ratliff et al., 2006a; Silver et al., 2008, 2010), manipulation (Ratliff et al., 2007), and quadruped locomotion (Ratliff et al., 2006a, 2007; Kolter et al., 2007).”
End-to-End Driving Via Conditional Imitation Learning (2017, background): “Imitation learning has been applied to a variety of tasks, including articulated motion [2], [28], [11], autonomous flight [1], [13], [30], modeling navigational behavior [37], [38], off-road driving [22], [32], and road following [4], [6], [27], [36].”
Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards (2017, background): “Imitation learning can be cast into a supervised learning problem (like classification) [10, 11].”
Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation (2017, background): “Behavioral cloning has led to many successes in robotics when a low-dimensional representation of environment state is available [31], [32], [33], [34].”
One-Shot Visual Imitation Learning via Meta-Learning (2017, background): “Efﬁcient imitation from a small number of demonstrations has been successful in scenarios where the state of the environment, such as the poses of objects, is known [8, 9, 10, 11].”
A Survey of Deep Learning Applications to Autonomous Vehicle Control (2019, background): “function from expert demonstrations of the task [180]–[183].”
Learning from Demonstrations: Is It Worth Estimating a Reward Function? (2013, methodology): “The classification algorithm is an algorithm using a structured large-margin approach [6].”
Contextual classification with functional Max-Margin Markov Networks (2009, methodology): “A variant of this algorithm is to perform exponentiated functional gradient descent, as described in [18].”
Toward fast policy search for learning legged locomotion (2012, background): “There have been a series of applications of machine learning approaches in legged locomotion such as Imitation Learning [20] and Reinforcement Learning (RL) [24], [22].”
Learning Deep Policies for Robot Bin Picking by Simulating Robust Grasping Sequences (2017, methodology): “Finally, we learn a CNN to classify the supervisor’s actions and use the trained CNN as action scoring function [33, 34].”
Learning to search: Functional gradient techniques for imitation learning (2009, methodology): “…from the behavior we desire (and is agnostic as to whether the demonstrating agent is truly trying to minimize a reward function in an MDP), provides approximation and regret (Ratliff et al., 2007a) guarantees on the resulting solutions, and admits a particular class of non-linear loss functions.”
Boosted and reward-regularized classification for apprenticeship learning (2014, methodology): “In [17], the authors use a large margin approach which is a score-based MCC where the decision rule g ∈ Y X is obtained via a score function q ∈ RX×Y such that ∀x ∈ X, g(x) ∈ argmaxy∈Y q(x, y).”
PI-QT-Opt: Predictive Information Improves Multi-Task Robotic Reinforcement Learning at Scale (2022, background): “Some approaches to real-world robot learning for manipulation utilize RL with real robot data collection or simulation with domain randomization [15, 29, 12, 13, 38, 39, 40, 41, 42, 43], while other approaches focus on imitation learning from expert demonstrations [27, 44, 45, 46, 47, 48].”
Parrot: Data-Driven Behavioral Priors for Reinforcement Learning (2020, methodology): “Our work is related to methods for learning from demonstrations (Pomerleau, 1989; Schaal et al., 2003; Ratliff et al., 2007; Pastor et al., 2009; Ho & Ermon, 2016; Finn et al., 2017b; Giusti et al., 2016; Sun et al., 2017; Zhang et al., 2017; Lynch et al., 2019).”
Multimodal End-to-End Autonomous Driving (2019, background): “There are two main approaches in the end-to-end driving paradigm, namely reinforcement learning [87], [88], [89] and imitation learning [90], [91].”
Time-Contrastive Networks: Self-Supervised Learning from Video (2017, methodology): “used for learning robotic skills from expert demonstrations [4, 5, 6, 7] and can be split into two areas: be-”
Watch this: Scalable cost-function learning for path planning in urban environments (2016, background): “alisation becomes important [5], [10]. Existing work in Inverse Reinforcement Learning shows the ease of directly deploying the learned cost maps with existing state-of-the-art planning systems [11], [12], [13], while current policy imitation based approaches [14], [15], [16] are of limited use in l”