Composable Deep Reinforcement Learning for Robotic Manipulation

By Tuomas Haarnoja, Vitchyr H. Pong, Aurick Zhou, Murtaza Dalal, Pieter Abbeel, Sergey Levine. 2018

254 citations recorded by the source corpus.

Model-free deep reinforcement learning has been shown to exhibit good performance in domains ranging from video games to simulated robotic manipulation and locomotion. However, model-free methods are known to perform poorly when the interaction time with the environment is limited, as is the case for most real-world robotic tasks. In this paper, we study how maximum entropy policies trained using soft Q-learning can be applied to real-world robotic manipulation. The application of this method to real-world manipulation is facilitated by two important features of soft Q-learning. First, soft Q-learning can learn multimodal exploration strategies by learning policies represented by expressive energy-based models. Second, we show that policies learned with soft Q-learning can be composed to create new policies, and that the optimality of the resulting policy can be bounded in terms of the divergence between the composed policies. This compositionality provides an especially valuable tool for real-world manipulation, where constructing new policies by composing existing skills can provide a large gain in efficiency over training from scratch. Our experimental evaluation demonstrates that soft Q-learning is substantially more sample efficient than prior model-free deep reinforcement learning methods, and that compositionality can be performed for both simulated and real-world tasks.

Subfields: Artificial Intelligence, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review (2018, methodology): “In another class of methods (Haarnoja et al., 2017, 2018a), the latent variable z is selected independently at each time step, and the policy π(a|s, z) has some simple unimodal form (e.g., a Gaussian distribution) conditioned on z, but a complex multimodal form when z is integrated out.”
Continual Learning of Control Primitives: Skill Discovery via Reset-Games (2020, background): “, Atari [29]), this is significantly more challenging in real-world physical tasks, as it requires human intervention in the form of manual resets or the availability of a hard coded reset [28, 47, 37, 52, 21].”
Composable Action-Conditioned Predictors: Flexible Off-Policy Learning for Robot Navigation (2018, neutral): “[36], but without entropy maximization.”
Soft Actor-Critic Algorithms and Applications (2018, background): “…where the optimal policy is independent of scaling of the reward function, in maximum entropy reinforcement learning the scaling factor has to be compensated by the choice a of suitable temperature, and a sub-optimal temperature can drastically degrade the performance (Haarnoja et al., 2018c).”
Modular Lifelong Reinforcement Learning via Neural Composition (2022, background): “One additional form of composition in RL uses full task policies as the components, and combines policies together when solving a new task (e.g., by summing their value functions; Todorov, 2009; Barreto et al., 2018; Haarnoja et al., 2018; Van Niekerk et al., 2019; Nangue Tasse et al., 2020).”
Constrained Exploration and Recovery from Experience Shaping (2018, background): “…for deep neural network architectures to tackle dif-ﬁcult control and decision-making problems, such as playing video games from raw pixel information (Mnih et al. 2015), Go (Silver et al. 2016), as well as robot manipulation (Haarnoja et al. 2018) and whole-body control (Peng et al. 2018).”
ManipulaTHOR: A Framework for Visual Object Manipulation (2021, background): “[18] propose a reinforcement learning approach for robotic manipulation where they construct new policies by composing existing skills.”
If MaxEnt RL is the Answer, What is the Question? (2019, background): “[16] Tuomas Haarnoja, Vitchyr Pong, Aurick Zhou, Murtaza Dalal, Pieter Abbeel, and Sergey Levine.”
How to train your robot with deep reinforcement learning: lessons we have learned (2021, methodology): “Although most model-free deep RL algorithms that have been applied to learn manipulation skills directly from real-world data have used off-policy algorithms based on Q-learning (Haarnoja et al. 2018a; Gu et al. 2017) or actor-critic designs (Haarnoja et al. 2018b), on-policy policy gradient algorit”
Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoning (2021, background): “Popular approaches to composing skills involve combining them in the space of value functions (Todorov, 2009; Hunt et al., 2019; da Silva et al., 2009; Ziebart, 2010; Haarnoja et al., 2018), and more recently, using pseudo-rewards (Barreto et al.”
The Option Keyboard: Combining Skills in Reinforcement Learning (2021, methodology): “In order to generate the polar scatter chart shown in Figure 4 we sampled 10 000 pairs ( s, w ) , with s sampled uniformly at random from S and abstract actions w sampled from an isotropic Gaussian distribution in R d with unit variance (where d = 3 for AVC and d = 2 for GPE and GPI).”
Time Reversal as Self-Supervision (2018, background): “Lego) block mating, owing to its close resemblance with industrial assembly applications, has been studied in the past [1] as it nicely casts the challenges of assembly in a set up that can be closely studied.”
Multi-Task Reinforcement Learning with Soft Modularization (2020, methodology): “The same multi-task SAC and multi-task multi-head SAC baselines are also proposed in (Yu et al., 2019).”
Explainability in Deep Reinforcement Learning (2020, background): “Moreover, when encompassing human advice during training, it can be derived into advisable learning which further improves user trust as the model can understand human natural language and yields clear and precise explanations [100].”
How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Composition (2022, background): “This intuition has driven theoretical and empirical results in the setting of entropy-regularized RL (Haarnoja et al., 2018; Van Niekerk et al., 2019).”
Active Domain Randomization (2019, methodology): “We set the total range of variation for the MES to be [8 , 20] (the default is 13, and lower than 7.5 makes the environment unsolv-able when all other physics parameters are held constant) and ﬁnd through empirical tests that lower engine strengths generate harder MDPs to solve.”
The Emergence of Individuality in Multi-Agent Reinforcement Learning (2020, methodology): “Maximum entropy methods [22, 23, 24] explore many possible ways to achieve a goal by maximizing the entropy of the policy.”
Learning to Coordinate Manipulation Skills via Skill Behavior Diversification (2020, methodology): “Maximum entropy policies (Haarnoja et al., 2017; 2018a;b) can learn diverse ways to achieve a goal by maximizing not only reward but also entropy of the policy.”
Learning to Walk via Deep Reinforcement Learning (2018, methodology): “Maximum entropy RL is both sample efficient and robust, making it a good candidate for real-world robot learning [16].”