DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction

By Aviral Kumar, Abhishek Gupta, Sergey Levine. 2020

122 citations recorded by the source corpus.

Deep reinforcement learning can learn effective policies for a wide range of tasks, but is notoriously difficult to use due to instability and sensitivity to hyperparameters. The reasons for this remain unclear. When using standard supervised methods (e.g., for bandits), on-policy data collection provides "hard negatives" that correct the model in precisely those states and actions that the policy is likely to visit. We call this phenomenon "corrective feedback." We show that bootstrapping-based Q-learning algorithms do not necessarily benefit from this corrective feedback, and training on the experience collected by the algorithm is not sufficient to correct errors in the Q-function. In fact, Q-learning and related methods can exhibit pathological interactions between the distribution of experience collected by the agent and the policy induced by training on that experience, leading to potential instability, sub-optimal convergence, and poor results when learning from noisy, sparse or delayed rewards. We demonstrate the existence of this problem, both theoretically and empirically. We then show that a specific correction to the data distribution can mitigate this issue. Based on these observations, we propose a new algorithm, DisCor, which computes an approximation to this optimal distribution and uses it to re-weight the transitions used for training, resulting in substantial improvements in a range of challenging RL settings, such as multi-task learning and learning from noisy reward signals. Blog post presenting a summary of this work is available at: this https URL.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Reinforcement Learning, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Reinforcement Learning

In Optimization & Training

Citations in the corpus

Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “A variant of this issue was noted in the standard RL setting, referred to as an absence of “corrective feedback” by Kumar et al. (2020) (see Kumar and Gupta (2020) for a short summary), and such a problem may affect offline RL algorithms with function approximation more severely, since they have no…”
Conservative Q-Learning for Offline Reinforcement Learning (2020, background): “[28] Aviral Kumar, Abhishek Gupta, and Sergey Levine.”
MOPO: Model-based Offline Policy Optimization (2020, background): “Model error on out-of-distribution states that often drives exploration and corrective feedback in the online setting [36] can be detrimental when interaction is not allowed.”
Regret Minimization Experience Replay in Off-Policy Reinforcement Learning (2021, methodology): “They are either based on TD error [9, 10, 12] or focused on the existence of corrective feedback [11].”
What are the Statistical Limits of Offline RL with Linear Function Approximation? (2020, background): “The condition that the optimal Q -function is only approximately linear is crucial for the correctness of the hardness result in Du et al. [2020] The techniques in [Du et al., 2020] are later generalized to other settings, including [Kumar et al., 2020, Wang et al., 2020, Mou et al., 2020].”