Boosting Soft Actor-Critic: Emphasizing Recent Experience without Forgetting the Past

By Che Wang, Keith W. Ross. 2019

54 citations recorded by the source corpus.

Soft Actor-Critic (SAC) is an off-policy actor-critic deep reinforcement learning (DRL) algorithm based on maximum entropy reinforcement learning. By combining off-policy updates with an actor-critic formulation, SAC achieves state-of-the-art performance on a range of continuous-action benchmark tasks, outperforming prior on-policy and off-policy methods. The off-policy method employed by SAC samples data uniformly from past experience when performing parameter updates. We propose Emphasizing Recent Experience (ERE), a simple but powerful off-policy sampling technique, which emphasizes recently observed data while not forgetting the past. The ERE algorithm samples more aggressively from recent experience, and also orders the updates to ensure that updates from old data do not overwrite updates from new data. We compare vanilla SAC and SAC+ERE, and show that ERE is more sample efficient than vanilla SAC for continuous-action Mujoco tasks. We also consider combining SAC with Priority Experience Replay (PER), a scheme originally proposed for deep Q-learning which prioritizes the data based on temporal-difference (TD) error. We show that SAC+PER can marginally improve the sample efficiency performance of SAC, but much less so than SAC+ERE. Finally, we propose an algorithm which integrates ERE and PER and show that this hybrid algorithm can give the best results for some of the Mujoco tasks.

Subfields: Artificial Intelligence, Computational Theory and Mathematics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

Citations in the corpus

Continual Reinforcement Learning with Multi-Timescale Replay (2020, background): “…a sequential multi-task setting [Isele and Cosgun, 2018, Rolnick et al., 2019], as well as in a single task setting [de Bruin et al., 2015, 2016, Zhang and Sutton, 2017, Wang and Ross, 2019], is that it is important to maintain a balance between the storage of new and old experiences in the buffer.”
Model-augmented Prioritized Experience Replay (2022, methodology): “Prioritized Experience Replay (PER) and its variants are one of the most frequently used methods to sample relatively important transitions from the replay buffer for Q - learning (Hessel et al., 2018; Schaul et al., 2016; Wang & Ross, 2019; Brittain et al., 2019).”
Experience Replay with Likelihood-free Importance Weights (2020, methodology): “Our method outperforms other approaches to weighted experience replay [32, 42] in terms of sample complexity over 4 out of 5 tasks with SAC and all tasks with TD3.”