Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
By Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, Sergey Levine. International Conference on Machine Learning, 2018
11,773 citations recorded by the source corpus.
Model-free deep reinforcement learning (RL) algorithms have been demonstrated on a range of challenging decision making and control tasks. However, these methods typically suffer from two major challenges: very high sample complexity and brittle convergence properties, which necessitate meticulous hyperparameter tuning. Both of these challenges severely limit the applicability of such methods to complex, real-world domains. In this paper, we propose soft actor-critic, an off-policy actor-critic deep RL algorithm based on the maximum entropy reinforcement learning framework. In this framework, the actor aims to maximize expected reward while also maximizing entropy. That is, to succeed at the task while acting as randomly as possible. Prior deep RL methods based on this framework have been formulated as Q-learning methods. By combining off-policy updates with a stable stochastic actor-critic formulation, our method achieves state-of-the-art performance on a range of continuous control benchmark tasks, outperforming prior on-policy and off-policy methods. Furthermore, we demonstrate that, in contrast to other off-policy algorithms, our approach is very stable, achieving very similar performance across different random seeds.
Subfields: Artificial Intelligence, Cognitive Neuroscience, Electrical and Electronic Engineering, Reinforcement Learning
View Paper: https://arxiv.org/abs/1801.01290
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Pieter Abbeel
- Yoshua Bengio
- Chelsea Finn
- Ruslan Salakhutdinov
- Mohit Bansal
- Quoc V. Le
- Trevor Darrell
In Cognitive Neuroscience
- Sergey Levine
- Chelsea Finn
- Pieter Abbeel
- Yoshua Bengio
- Shimon Whiteson
- Aviral Kumar
- Abhishek Gupta
- Olivier Pietquin
In Electrical and Electronic Engineering
- Sergey Levine
- Ofir Nachum
- Aviral Kumar
- Shixiang Shane Gu
- George Tucker
- Chelsea Finn
- Shimon Whiteson
- Pieter Abbeel
In Reinforcement Learning
- Sergey Levine
- Pieter Abbeel
- Chelsea Finn
- Ofir Nachum
- Shimon Whiteson
- Shixiang Shane Gu
- Aviral Kumar
- George Tucker
Citations in the corpus
- Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog (2019, neutral): “[22, 32, 44, 48]).”
- On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgetting (2022, background): “, 2017) and SAC (Haarnoja et al., 2018) the target distribution is defined as π∗(a|s) = exp(Qθ(s, a))/Zθ(s), where Q is a state-action value function and Z is a partition function of for a given state, both dependent on policy parameters θ.”
- Human-centric Dialog Training via Offline Reinforcement Learning (2020, methodology): “Well-known examples include Trust Region Policy Optimization (TRPO) (Schulman et al., 2015), and use conservative, KL-regularized policy updates to restrict the RL algorithm to stay close to its own prior policy (Haarnoja et al., 2018; Kakade, 2002; Peters et al., 2010; Rawlik et al., 2012).”
- Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning (2019, methodology): “DDPG, TD3, and SAC are not easily applicable to discrete action spaces due to their need to back-propagate from the Q-function to the policy.”
- Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review (2018, methodology): “In another class of methods (Haarnoja et al., 2017, 2018a), the latent variable z is selected independently at each time step, and the policy π(a|s, z) has some simple unimodal form (e.g., a Gaussian distribution) conditioned on z, but a complex multimodal form when z is integrated out.”
- The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models (2022, methodology): “We use PPO (Schulman et al., 2017) to optimize policies for the traffic and COVID environments, SAC (Haarnoja et al., 2018) to optimize the policies for the glucose environment, and torch-beast (K¨uttler et al., 2019), a PyTorch implementation of IMPALA (Espeholt et al., 2018), to optimize the…”
- Addressing Function Approximation Error in Actor-Critic Methods (2018, methodology): “, 2017), and SAC (Haarnoja et al., 2018), as implemented by the author’s GitHub1.”
- Off-Policy Actor-Critic with Shared Experience Replay (2019, background): “Hence, most established actor-critic algorithms with replay such as (Wang et al., 2017; Gruslys et al., 2018; Haarnoja et al., 2018) employ and maintain Q-functions to learn from the replayed off-policy experience.”
- Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “Figure 2: Performance of SAC (Haarnoja et al., 2018), an actor-critic method, on the HalfCheetah-v2 task in the offline setting, showing return as a function of gradient steps (left) and average learned Q-values on a log scale (right), for different numbers of training points (n).”
- Accelerating Online Reinforcement Learning with Offline Datasets (2020, methodology): “We utilize a standard temporal difference learning scheme for policy evaluation [7, 4], by minimizing the Bellman error.”
- The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games (2021, methodology): “In single-agent continuous control tasks [8], advances in off-policy methods such as SAC [13] led to a consensus that despite their early success, policy gradient (PG) algorithms such as PPO are less sample efficient than off-policy methods.”
- CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learning (2020, methodology): “2), we train agents under the three described curricula using the following MF-RL algorithms: The original Proximal Policy Optimization (PPO) from Schulman et al. (2017), Soft Actor-Critic (SAC) from Haarnoja et al. (2018) and the Twin Delayed DDPG (TD3) from Fujimoto et al. (2018).”
- Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models (2018, methodology): “25 times fewer data on half-cheetah compared to the model-free Soft Actor Critic algorithm [15].”
- Disentangling Transfer in Continual Reinforcement Learning (2022, methodology): “In our study, we focus on soft actor-critic (SAC) [15], an off-policy actor-critic RL algorithm, based on the maximum entropy principle.”
- CoMPS: Continual Meta Policy Search (2021, methodology): “PEARL generally performs poorly across all tasks: although SAC is an off-policy algorithm, it is well known that such methods do not perform well when they are not allowed to gather any additional online data (as, for example, in the case of offline RL) (Kumar et al., 2019).”
- Same State, Different Task: Continual Reinforcement Learning without Interference (2021, methodology): “We have continuous actions and train a multihead policy and Q-function using Soft Actor Critic (Haarnoja et al. 2018b).”
- Continual World: A Robotic Benchmark For Continual Reinforcement Learning (2021, methodology): “SAC is an off-policy algorithm using replay buffer, which is an important aspect for CL, particularly for methods relying on rehearsing old trajectories.”
- Continual Reinforcement Learning with Multi-Timescale Replay (2020, methodology): “We used the automatic entropy regulariser used in [Haarnoja et al., 2018b], which was found to be more robust than a fixed entropy regularisation coefficient.”