Residual Reinforcement Learning for Robot Control

By Tobias Johannink, Shikhar Bahl, Ashvin Nair, Jianlan Luo, Avinash Kumar, Matthias Loskyll, Juan Aparicio Ojea, Eugen Solowjow, Sergey Levine. 2018

624 citations recorded by the source corpus.

Conventional feedback control methods can solve various types of robot control problems very efficiently by capturing the structure with explicit models, such as rigid body equations of motion. However, many control problems in modern manufacturing deal with contacts and friction, which are difficult to capture with first-order physical modeling. Hence, applying control design methodologies to these kinds of problems often results in brittle and inaccurate controllers, which have to be manually tuned for deployment. Reinforcement learning (RL) methods have been demonstrated to be capable of learning continuous robot controllers from interactions with the environment, even for problems that include friction and contacts. In this paper, we study how we can solve difficult control problems in the real world by decomposing them into a part that is solved efficiently by conventional feedback control methods, and the residual which is solved with RL. The final control policy is a superposition of both control signals. We demonstrate our approach by training an agent to successfully perform a real-world block assembly task involving contacts and unstable objects.

Subfields: Artificial Intelligence, Biomedical Engineering, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

In Control and Systems Engineering

Citations in the corpus

Off-Policy Deep Reinforcement Learning without Exploration (2018, background): “Our work also connects to residual policy learning (Johannink et al., 2018; Silver et al., 2018), where the initial policy is the generative model, rather than an expert or feedback controller.”
SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulation (2020, background): “Many of these benchmarks assume that the agent directly observes a low dimension state representation that fully describes the underlying dynamics of the environment, such as the joint angles and velocities of the robot [13, 16] or object state [8, 17].”
Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Text (2020, background): “Note that there is no reason, in theory at least, why the recipe could not apply to robot-learning for object manipulation (e.g. Johannink et al. [2019]).”
TossingBot: Learning to Throw Arbitrary Objects With Residual Physics (2019, background): “Concurrent work on residual reinforcement learning [12, 25] investigates the benefits of residual control under variation in control and sensor noise, partial observability, and transfer from sim-to-real.”
How to train your robot with deep reinforcement learning: lessons we have learned (2021, background): “3 Joint Training Instead of simply pre-training the policy with supervised learning, we can train it jointly, adding together the loss from the policy gradient objective with the loss for behavioral cloning (Hester et al. 2018; Wu et al. 2019; Johannink et al. 2019).”
Continual Semantic Segmentation with Automatic Memory Sample Selection (2023, background): “Reinforcement learning (RL) has achieved remarkable success in many decision-making tasks like game intelligence [41] and robot control [24, 27].”
Residual Policy Learning (2018, background): “Concurrent work by Johannink et al. [28] also proposes residual reinforcement learning, and focuses on showing the value of the approach for real robots in a task of block insertion, investigating the effects of variation in the initial state, control noise, and the transfer from sim to real.”
Residual Policy Learning for Shared Autonomy (2020, methodology): “Our work builds off of a recent advance in reinforcement learning for robotics: residual policy learning [26, 47].”
Reinforcement Learning for Robust Parameterized Locomotion Control of Bipedal Robots (2021, methodology): “In [4], [5], reference motions combined with model-free residual learning [6] are used to learn walking policies that are able to reliably track given planar velocity”
Side-Tuning: A Baseline for Network Adaptation via Additive Side Networks (2019, background): “Residual learning has also been explored in robotics as residual RL [10,33], in which we train an agent for a single task by first taking a coarse policy (e.”
Shaping Rewards for Reinforcement Learning with Imperfect Demonstrations using Generative Models (2020, positive): “Residual RL: Methods in this family [35], [36], [37] decompose the policy into a sum of two parts, one representing prior knowledge, namely trained from demonstration data, and one residual policy that is learned through RL.”
Learning Visual Affordances for Robotic Manipulation (2019, background): “These benefits are also observed in concurrent work on residual reinforcement learning [54, 108] in block-assembly and object manipulation tasks.”
Side-Tuning: Network Adaptation via Additive Side Networks (2019, methodology): “This has been successfully used in computer vision (ResNets [8]) and in robotics, where residual RL [10, 30] (ii) shows the connectivity structure when using side-tuning along with alpha-blending.”
Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes (2024, background): “A common strategy among approaches to assembly is using residual RL (126), in which a residual policy is learned on top of a reference trajectory.”
Parrot: Data-Driven Behavioral Priors for Reinforcement Learning (2020, background): “While demonstrations can also be used to speed up RL (Schaal, 1996; Peters & Schaal, 2006; Kormushev et al., 2010; Hester et al., 2017; Vecerı́k et al., 2017; Nair et al., 2018; Rajeswaran et al., 2018; Silver et al., 2018; Peng et al., 2018; Johannink et al., 2019; Gupta et al., 2019), this usually”