Training Agents using Upside-Down Reinforcement Learning

By Rupesh Kumar Srivastava, Pranav Shyam, Filipe Wall Mutz, Wojciech Jaśkowski, J. Urgen Schmidhuber. 2019

140 citations recorded by the source corpus.

Traditional Reinforcement Learning (RL) algorithms either predict rewards with value functions or maximize them using policy search. We study an alternative: Upside-Down Reinforcement Learning (Upside-Down RL or UDRL), that solves RL problems primarily using supervised learning techniques. Many of its main principles are outlined in a companion report [34]. Here we present the first concrete implementation of UDRL and demonstrate its feasibility on certain episodic learning problems. Experimental results show that its performance can be surprisingly competitive with, and even exceed that of traditional baseline algorithms developed over decades of research.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

Teaching Autoregressive Language Models Complex Tasks By Demonstration (2021, background): “…sequences of states and actions is also reminiscent of recent formulations of reinforcement learning as a sequence modeling problem involving undifferentiated sequences of states, actions, and rewards; this setup can also be deployed in an imitation learning context [17], [18] (see also [25]).”
Decision Transformer: Reinforcement Learning via Sequence Modeling (2021, methodology): “Recent work [43, 44, 45] studies “upside-down” reinforcement learning (UDRL), which are similar to our method in seeking to model behaviors with a supervised loss conditioned on the target return.”
Offline Reinforcement Learning as One Big Sequence Modeling Problem (2021, background): “…work on goal relabeling (Andrychowicz et al., 2017; Rauber et al., 2019; Ghosh et al., 2021; Paster et al., 2021) and reward conditioning (Schmidhuber, 2019; Srivastava et al., 2019; Kumar et al., 2019b) to reinterpret all past experience as useful demonstrations with proper contextualization.”
RUDDER: Return Decomposition for Delayed Rewards (2018, background): ““Hindsight Credit Assignment” (HCA) [17] and “Upside-Down Reinforcement Learning” [39, 45] use supervised learning via cross-entropy to select a backward model that predicts the action to take (output) to achieve a desired return (input 1) from current state (input 2) in a certain number of steps (i”
Can Wikipedia Help Offline Reinforcement Learning? (2022, methodology): “Building on reward-conditioned imitation learning (Srivastava et al., 2019; Kumar et al., 2019), Transformer architecture has been recently adopted for re-placing ofﬂine RL with sequence modeling (Chen et al., 2021; Janner et al., 2021; Furuta et al., 2021).”
RvS: What is Essential for Offline RL via Supervised Learning? (2021, methodology): “By conditioning on some piece of information, such as a goal, reward function parameterization, or reward value, such experience can be used for simple behavior cloning [5, 6, 8, 10, 17, 23, 26, 29, 32, 34, 38].”
Is Conditional Generative Modeling all you need for Decision-Making? (2022, background): “Reward Conditioned Policies Prior works (Kumar et al., 2019; Schmidhuber, 2019; Emmons et al., 2021; Chen et al., 2021) have studied learning of reward conditioned policies via reward conditioned behavioral cloning.”
Planning from Pixels using Inverse Dynamics Models (2020, background): “Speciﬁcally, rather than learn to map states and actions to reward, as is typical in reinforcement learning, Srivastava et al. (2019) train a model to predict actions given a state and an outcome, which could be the amount of reward the agent is to collect within a certain amount of time.”
Reinforcement Learning as One Big Sequence Modeling Problem (2021, methodology): “In the context of model-free RL, our method also resembles recently proposed work on goal relabeling [2, 48, 18] and reward-conditioning [52, 55, 32] to reinterpret all past experience as useful demonstrations with proper contextualization.”
Reinforcement Learning Upside Down: Don't Predict Rewards - Just Map Them to Actions (2019, methodology): “asics of RL for deterministic environments and Markovian interfaces between controller and environment (Sec. 3), then proceed to more complexcases in a series of additional Sections. A separate paper [63] describes the concrete RL implementations used in our ﬁrst experiments with RL, and presents re”
Simplifying Deep Reinforcement Learning via Self-Supervision (2021, methodology): “Upside-Down Reinforcement Learning (UDRL) As far as we know, there is no ofﬁcial implementation.”
Contrastive Learning as Goal-Conditioned Reinforcement Learning (2022, methodology): “Note that setting λ = 1 corresponds to GCBC [15, 21, 24, 40, 83, 96, 116, 119], which we will include as a baseline.”