Soft Actor-Critic Algorithms and Applications
By Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel and 1 more. 2018
3,200 citations recorded by the source corpus.
Model-free deep reinforcement learning (RL) algorithms have been successfully applied to a range of challenging sequential decision making and control tasks. However, these methods typically suffer from two major challenges: high sample complexity and brittleness to hyperparameters. Both of these challenges limit the applicability of such methods to real-world domains. In this paper, we describe Soft Actor-Critic (SAC), our recently introduced off-policy actor-critic algorithm based on the maximum entropy RL framework. In this framework, the actor aims to simultaneously maximize expected return and entropy. That is, to succeed at the task while acting as randomly as possible. We extend SAC to incorporate a number of modifications that accelerate training and improve stability with respect to the hyperparameters, including a constrained formulation that automatically tunes the temperature hyperparameter. We systematically evaluate SAC on a range of benchmark tasks, as well as real-world challenging tasks such as locomotion for a quadrupedal robot and robotic manipulation with a dexterous hand. With these improvements, SAC achieves state-of-the-art performance, outperforming prior on-policy and off-policy methods in sample-efficiency and asymptotic performance. Furthermore, we demonstrate that, in contrast to other off-policy algorithms, our approach is very stable, achieving similar performance across different random seeds. These results suggest that SAC is a promising candidate for learning in real-world robotics tasks.
Subfields: Artificial Intelligence, Reinforcement Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Pieter Abbeel
- Chelsea Finn
- Yoshua Bengio
- Ruslan Salakhutdinov
- Mohit Bansal
- Trevor Darrell
- Quoc V. Le
In Reinforcement Learning
- Sergey Levine
- Pieter Abbeel
- Chelsea Finn
- Ofir Nachum
- Shixiang Shane Gu
- Aviral Kumar
- Abhishek Gupta
- George Tucker
Citations in the corpus
- Solving Rubik's Cube with a Robot Hand (2019, background): “Soft Actor-Critic[40] has been recently proposed as a state-of-the-art model-free algorithm optimizing concurrently both expected reward and action entropy, that is capable of learning complex behaviors directly in the real world.”
- Deep Reinforcement Learning for Autonomous Driving: A Survey (2020, methodology): “Two methods with common authors are energy-based policies [30] and more recent and with widespread use, the Soft Actor Critic (SAC) algorithm [31], both rely on adding an entropy term to the reward function, so we update the policy objective from Eqn.”
- Disentangling Transfer in Continual Reinforcement Learning (2022, methodology): “The results are presented in Tables 15, 16, 17, indicating that direct regularization of the critic does not significantly improve the performance.”
- Same State, Different Task: Continual Reinforcement Learning without Interference (2021, methodology): “We have continuous actions and train a multihead policy and Q-function using Soft Actor Critic (Haarnoja et al. 2018b).”
- Continual World: A Robotic Benchmark For Continual Reinforcement Learning (2021, methodology): “[17] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine.”
- Continual Reinforcement Learning with Multi-Timescale Replay (2020, methodology): “We used the automatic entropy regulariser used in [Haarnoja et al., 2018b], which was found to be more robust than a fixed entropy regularisation coefficient.”
- Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning (2020, background): “…also bears similarity to previous works that use (relative) entropy regularized policy updates which implement constraints with respect to either a fixed (e.g. uniform) policy (e.g. Haarnoja et al., 2018a) or, in a trust-region like scheme, to the previous policy (e.g. Abdolmaleki et al., 2018).”
- Towards Characterizing Divergence in Deep Q-Learning (2019, methodology): “Our implementation of PreQN was significantly slower than our implementation of SAC (by more than 50%), due to the requirement of calculating backward passes separately for each state-action pair in the batch, and solving the system of equations K θ Z = ∆ .”
- Offline Reinforcement Learning with Fisher Divergence Critic Regularization (2021, methodology): “Thus, we propose to train our actor πφ exactly in this manner (which is already popular in modelfree actor critic algorithms (Haarnoja et al., 2019)), and then use it in eq.”
- Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics (2020, methodology): “The path from SAC (Haarnoja et al., 2018b) to TQC comprises five modifications: Q-network size increase ( B ig), quantile Q-network introduction ( Q uantile), target distribution truncation ( T runcate), atom pooling ( P ool), and ensem-bling.”
- Hyperparameter Selection for Imitation Learning (2021, methodology): “Both, AIL and PWIL use SAC (Haarnoja et al., 2018a) with an entropy constraint (Haarnoja et al., 2018b) to train the policy and optionally use absorbing states introduced in DAC (Kostrikov et al., 2018).”
- Representation Matters: Offline Pretraining for Sequential Decision Making (2021, background): “SAC is a well-performing algorithm on its own, and so to increase the difficulty of the downstream task, we consider a simple modification to make our domains partially observable: zero-masking out a random dimension of the state observation.”
- Accelerating Reinforcement Learning with Learned Skill Priors (2020, methodology): “[11] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al.”
- robosuite: A Modular Simulation Framework and Benchmark for Robot Learning (2020, methodology): “The SAC algorithm is able to solve three of the nine environments, including Block Lifting, Door Opening, and Two Arm Peg-inHole, while making slow progress in the other environments, which requires intelligent exploration in longer task horizons.”
- What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study (2020, methodology): “For continuous control tasks, on-policy algorithms such as REINFORCE [2], TRPO [10], A3C [14], PPO [17] and off-policy algorithms such as DDPG [13] and SAC [21] have enabled successful applications such as quadrupedal locomotion [20], self-driving [30] or dexterous in-hand manipulation [20, 25, 32].”
- Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels (2020, methodology): “(iii) Combined with vanilla SAC (Haarnoja et al., 2018) and using hyper-parameters fixed across all tasks, the overall approach obtains state-of-the-art performance on the DeepMind control suite (Tassa et al.”
- End-to-End Robotic Reinforcement Learning without Reward Engineering (2019, methodology): “We implemented our methods on top of a standard opensource implementation of the soft actor-critic algorithm [37].”