Reward-Conditioned Policies

By Aviral Kumar, Xue Bin Peng, Sergey Levine. 2019

117 citations recorded by the source corpus.

Reinforcement learning offers the promise of automating the acquisition of complex behavioral skills. However, compared to commonly used and well-understood supervised learning methods, reinforcement learning algorithms can be brittle, difficult to use and tune, and sensitive to seemingly innocuous implementation decisions. In contrast, imitation learning utilizes standard and well-understood supervised learning methods, but requires near-optimal expert data. Can we learn effective policies via supervised learning without demonstrations? The main idea that we explore in this work is that non-expert trajectories collected from sub-optimal policies can be viewed as optimal supervision, not for maximizing the reward, but for matching the reward of the given trajectory. By then conditioning the policy on the numerical value of the reward, we can obtain a policy that generalizes to larger returns. We show how such an approach can be derived as a principled method for policy search, discuss several variants, and compare the method experimentally to a variety of current reinforcement learning methods on standard benchmarks.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

Pretraining Language Models with Human Preferences (2023, background): “Conditional training has recently emerged as an effective apporoach to offline RL (Schmidhuber, 2019; Kumar et al., 2019) and demonstrated strong results when paired with transformers (Chen et al., 2021a; Janner et al., 2021).”
Offline Reinforcement Learning as One Big Sequence Modeling Problem (2021, methodology): “…that serve to constrain the learned policy to avoid out-of-2 distribution behavior that is not supported under the dataset (Fujimoto et al., 2019; Kumar et al., 2019a; Ghasemipour et al., 2021); imitation learning, where it is used to ﬁt an expert’s actions to obtain a policy (Ross & Bagnell,…”
Critic Regularized Regression (2020, neutral): “Last but not least, methods that rely on reward ﬁltering would also favor arm one over arm two [7, 18].”
Training Agents using Upside-Down Reinforcement Learning (2019, methodology): “We briefly discuss an inconsistency between training and rollout formulations in RCP-R [Kumar et al., 2019].”
Can Wikipedia Help Offline Reinforcement Learning? (2022, methodology): “In addition to benchmarking our pre-trained language models, we compare to popular state-of-the-art of-ﬂine RL algorithms as follows: Decision Transformer (DT) (Chen et al., 2021), CQL (Kumar et al., 2020), TD3+BC (Fujimoto & Gu, 2021), BRAC (Wu et al., 2019), and AWR baselines (Peng et al., 2019).”
Reinforcement Learning as One Big Sequence Modeling Problem (2021, methodology): “In the context of model-free RL, our method also resembles recently proposed work on goal relabeling [2, 48, 18] and reward-conditioning [52, 55, 32] to reinterpret all past experience as useful demonstrations with proper contextualization.”
You Can't Count on Luck: Why Decision Transformers and RvS Fail in Stochastic Environments (2022, unknown): “A recent trend in ofﬂine RL has been to use simple approaches that do RL via Supervised Learning (RvS) (e.g., [1, 2, 6–8]) rather than typical value-based approaches.”
Is Conditional Generative Modeling all you need for Decision-Making? (2022, background): “Reward Conditioned Policies Prior works (Kumar et al., 2019; Schmidhuber, 2019; Emmons et al., 2021; Chen et al., 2021) have studied learning of reward conditioned policies via reward conditioned behavioral cloning.”