Reinforcement Learning Upside Down: Don't Predict Rewards - Just Map Them to Actions

By J. Urgen Schmidhuber. 2019

154 citations recorded by the source corpus.

We transform reinforcement learning (RL) into a form of supervised learning (SL) by turning traditional RL on its head, calling this Upside Down RL (UDRL). Standard RL predicts rewards, while UDRL instead uses rewards as task-defining inputs, together with representations of time horizons and other computable functions of historic and desired future data. UDRL learns to interpret these input observations as commands, mapping them to actions (or action probabilities) through SL on past (possibly accidental) experience. UDRL generalizes to achieve high rewards or other goals, through input commands such as: get lots of reward within at most so much time! A separate paper [63] on first experiments with UDRL shows that even a pilot version of UDRL can outperform traditional baseline algorithms on certain challenging RL problems. We also also conceptually simplify an approach [60] for teaching a robot to imitate humans. First videotape humans imitating the robot's current behaviors, then let the robot learn through SL to map the videos (as input commands) to these behaviors, then let it generalize and imitate videos of humans executing previously unknown behavior. This Imitate-Imitator concept may actually explain why biological evolution has resulted in parents who imitate the babbling of their babies.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Reinforcement Learning

Citations in the corpus

Offline Reinforcement Learning as One Big Sequence Modeling Problem (2021, background): “, 2021) and reward conditioning (Schmidhuber, 2019; Srivastava et al., 2019; Kumar et al., 2019b) to reinterpret all past experience as useful demonstrations with proper contextualization.”
RUDDER: Return Decomposition for Delayed Rewards (2018, background): ““Hindsight Credit Assignment” (HCA) [17] and “Upside-Down Reinforcement Learning” [39, 45] use supervised learning via cross-entropy to select a backward model that predicts the action to take (output) to achieve a desired return (input 1) from current state (input 2) in a certain number of steps (i”
Reward-Conditioned Policies (2019, neutral): “As noted in Section 2, concurrently to our work, Schmidhuber [37] proposed a similar approach, UDRL, though without weighting or advantage conditioning, and reports a ﬁnal result of around 150 on the LunarLander-v2 task.”
Training Agents using Upside-Down Reinforcement Learning (2019, background): “Many of its main principles are outlined in a companion report [34].”
The alignment problem from a deep learning perspective (2022, unknown): “…such as model-based RL and planning [Sutton and Barto, 2018] (with learned reward functions), goal-conditioned sequence modeling [Chen et al., 2021, Li et al., 2022, Schmidhuber, 2020], or RL on rewards learned via inverse RL [Ng and Russell, 2000]—however, these are beyond our current scope.”
Reinforcement Learning as One Big Sequence Modeling Problem (2021, methodology): “In the context of model-free RL, our method also resembles recently proposed work on goal relabeling [2, 48, 18] and reward-conditioning [52, 55, 32] to reinterpret all past experience as useful demonstrations with proper contextualization.”
Is Conditional Generative Modeling all you need for Decision-Making? (2022, background): “Reward Conditioned Policies Prior works (Kumar et al., 2019; Schmidhuber, 2019; Emmons et al., 2021; Chen et al., 2021) have studied learning of reward conditioned policies via reward conditioned behavioral cloning.”
GFlowNet Foundations (2021, unknown): “, 2018) or the rewardconditioned upside-down RL (Schmidhuber, 2019).”
Planning from Pixels using Inverse Dynamics Models (2020, background): “Recently, there has been an emergence of work (e.g., Ghosh et al., 2020; Schmidhuber, 2019; Srivas-tava et al., 2019) highlighting the relationship between imitation learning and reinforcement learning.”