Experience Replay with Likelihood-free Importance Weights

By Samarth Sinha, Jiaming Song, Animesh Garg, Stefano Ermon. 2020

74 citations recorded by the source corpus.

The use of past experiences to accelerate temporal difference (TD) learning of value functions, or experience replay, is a key component in deep reinforcement learning. Prioritization or reweighting of important experiences has shown to improve performance of TD learning this http URL this work, we propose to reweight experiences based on their likelihood under the stationary distribution of the current policy. Using the corresponding reweighted TD objective, we implicitly encourage small approximation errors on the value function over frequently encountered states. We use a likelihood-free density ratio estimator over the replay buffer to assign the prioritization weights. We apply the proposed approach empirically on two competitive methods, Soft Actor Critic (SAC) and Twin Delayed Deep Deterministic policy gradient (TD3) -- over a suite of OpenAI gym tasks and achieve superior sample complexity compared to other baseline approaches.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Sung Ju Hwang
2. Jaehong Yoon
3. Eunho Yang
4. Divyam Madaan
5. Samarth Sinha
6. Animesh Garg
7. Jing-Cheng Pang
8. Jinwoo Shin

In Computational Theory and Mathematics

1. Eunho Yang
2. Sung Ju Hwang
3. Animesh Garg
4. Samarth Sinha
5. Jinwoo Shin
6. Youngmin Oh

In Electrical and Electronic Engineering

1. Eunho Yang
2. Sung Ju Hwang
3. Animesh Garg
4. Samarth Sinha
5. Jinwoo Shin
6. Youngmin Oh
7. Zhenghai Xue
8. Shengyi Jiang

Citations in the corpus

• Online Coreset Selection for Rehearsal-based Continual Learning (2021, positive): “Importance sampling [13, 14, 38] strengthens the loss/gradients of important samples based on influence functions.”
• Model-augmented Prioritized Experience Replay (2022, methodology): “The solid lines and shaded regions denote the mean and standard deviations over the five random runs. strategies (Zha et al., 2019; Sinha et al., 2020; Oh et al., 2021) utilizing various features, including transition’s elements and quantities driven by critic networks.”
• Regret Minimization Experience Replay in Off-Policy Reinforcement Learning (2021, methodology): “They are either based on TD error [9, 10, 12] or focused on the existence of corrective feedback [11].”
• S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning (2021, methodology): “These constrains include using KL-divergence [10, 20], MMD [8] or f -divergence [21].”