End-to-End Robotic Reinforcement Learning without Reward Engineering

By Avi Singh, Larry Yang, Kristian Hartikainen, Chelsea Finn, Sergey Levine. 2019

289 citations recorded by the source corpus.

The combination of deep neural network models and reinforcement learning algorithms can make it possible to learn policies for robotic behaviors that directly read in raw sensory inputs, such as camera images, effectively subsuming both estimation and control into one model. However, real-world applications of reinforcement learning must specify the goal of the task by means of a manually programmed reward function, which in practice requires either designing the very same perception pipeline that end-to-end reinforcement learning promises to avoid, or else instrumenting the environment with additional sensors to determine if the task has been performed successfully. In this paper, we propose an approach for removing the need for manual engineering of reward specifications by enabling a robot to learn from a modest number of examples of successful outcomes, followed by actively solicited queries, where the robot shows the user a state and asks for a label to determine whether that state represents successful completion of the task. While requesting labels for every single state would amount to asking the user to manually provide the reward signal, our method requires labels for only a tiny fraction of the states seen during training, making it an efficient and practical approach for learning skills without manually engineered rewards. We evaluate our method on real-world robotic manipulation tasks where the observations consist of images viewed by the robot's camera. In our experiments, our method effectively learns to arrange objects, place books, and drape cloth, directly from images and without any manually specified reward functions, and with only 1-4 hours of interaction with the real world.

Subfields: Artificial Intelligence, Control and Systems Engineering, Reinforcement Learning, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Reinforcement Learning

In Robotics & Control

Citations in the corpus

A Framework for Data-Driven Robotics (2019, background): “Binary success labels of frames are also used to learn reward functions [26], [60].”
Human-in-the-Loop Imitation Learning using Remote Teleoperation (2020, background): “One paradigm is Reinforcement Learning (RL) with human feedback [40], where a human provides rewards during agent training [8, 11, 24, 25, 34, 36], but this suffers from the same limitations as IRL due to the need for extensive agent interaction.”
Language Conditioned Imitation Learning Over Unstructured Data (2020, background): “Reward design is non-trivial in complex environments, often requiring either task-speciﬁc instrumentation [13] or learned perceptual rewards [41, 42].”
Task-Relevant Adversarial Imitation Learning (2019, methodology): “To apply RL, it is essential that we have access to the reward function which is often hard to design and evaluate [Singh et al., 2019].”
XIRL: Cross-embodiment Inverse Reinforcement Learning (2021, methodology): “In [17], a small sub-set of states is human labeled for goal success and a 355 convolutional network is then trained to detect successful task completion from image observations, where 356 on-policy samples are used as negatives for the classifier.”
Hierarchical Foresight: Self-Supervised Learning of Long-Horizon Tasks via Visual Subgoal Generation (2019, background): “…Levine et al., 2015; Kalashnikov et al., 2018; Lange et al., 2012; OpenAI et al., 2018; Schenck & Fox, 2016; Matas et al., 2018; James et al., 2017; Singh et al., 2019), including in goal-conditioned settings (Kaelbling, 1993; Schaul et al., 2015; Andrychowicz et al., 2017; Sadeghi et al., 2018;…”
If MaxEnt RL is the Answer, What is the Question? (2019, background): “Empirically, MaxEnt RL algorithms achieve good performance on a number of simulated [17] and real-world [16, 46] control tasks, and can be more robust to perturbations [18].”
Broadly-Exploring, Local-Policy Trees for Long-Horizon Task Planning (2020, methodology): “Our work also relies on task success detectors, real-world experiments could make use of learned success classifiers such as [43].”
Self-Supervised Learning for Precise Pick-and-Place Without Object Model (2020, methodology): “[21] has learned robot manipulation in the context of Inverse RL.”
Example-Driven Model-Based Reinforcement Learning for Solving Long-Horizon Visuomotor Tasks (2021, background): “Model-free deep RL has been effective on short-horizon skills like object grasping [21, 20, 22], pushing and throwing [19, 23, 24], and multi-task learning [23, 25] from images.”
PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training (2021, background): “However, the common approach is to perform goal-conditioned learning with classiﬁers (Singh et al., 2019), which can only capture limited information about what goal states are , and not about how they can be achieved.”
Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation (2021, result): “Lastly, unlike other works which use classifiers for single-task reward learning [68, 69, 53] on robots, a languageconditioned reward classifier can flexibly represent many tasks with an easy to provide form of task-specification.”
COG: Connecting New Skills to Past Experience with Offline Reinforcement Learning (2020, background): “Robotic RL. RL has been applied to a wide variety of robotic manipulation tasks, including grasping objects [11, 12], in-hand object manipulation [13, 14, 15, 16], pouring ﬂuids [17], door opening [18, 19], and manipulating cloth [20, 21].”
VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training (2022, background): “Finally, we comment that whereas sparse reward still requires human engineering via installing additional sensors (Rajeswar et al., 2021; Singh et al., 2019) and faces exploration challenges (Nair et al., 2018), with VIP, the end-user has to provide only a goal image.”
Zero-Shot Reward Specification via Grounded Natural Language (2022, background): “Previous efforts have explored image-based goal specification, with significant successes in visual navigation and manipulation tasks (Pathak et al., 2018; Nair et al., 2018; Fu et al., 2018; Singh et al., 2019).”
Offline Reinforcement Learning from Images with Latent Space Models (2020, methodology): “…this problem with model-free RL methods by learning policies either from pixel inputs end-to-end (Haarnoja et al., 2018; Gelada et al., 2019; Singh et al., 2019; Kostrikov et al., 2020; Laskin et al., 2020; Han et al., 2020) or on top of unsupervised visual representations Lange and…”