Recall Traces: Backtracking Models for Efficient Reinforcement Learning

By Anirudh Goyal, Philemon Brakel, William Fedus, Soumye Singhal, Timothy P. Lillicrap, Sergey Levine, Hugo Larochelle, Yoshua Bengio. 2018

69 citations recorded by the source corpus.

In many environments only a tiny subset of all states yield high reward. In these cases, few of the interactions with the environment provide a relevant learning signal. Hence, we may want to preferentially train on those high-reward states and the probable trajectories leading to them. To this end, we advocate for the use of a backtracking model that predicts the preceding states that terminate at a given high-reward state. We can train a model which, starting from a high value state (or one that is estimated to have high value), predicts and sample for which the (state, action)-tuples may have led to that high value state. These traces of (state, action) pairs, which we refer to as Recall Traces, sampled from this backtracking model starting from a high value state, are informative as they terminate in good states, and hence we can use these traces to improve a policy. We provide a variational interpretation for this idea and a practical algorithm in which the backtracking model samples from an approximate posterior distribution over trajectories which lead to large rewards. Our method improves the sample efficiency of both on- and off-policy RL algorithms across several environments and tasks.

Subfields: Artificial Intelligence, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

Citations in the corpus

RUDDER: Return Decomposition for Delayed Rewards (2018, background): “Examples of learning a separate model are to trace back from known goal states [23] or from high reward states [36].”
Sample-Efficient Reinforcement Learning via Counterfactual-Based Data Augmentation (2020, background): “Backtracking models [34] consider predicting the samples that may lead to high-reward states.”
The Effect of Planning Shape on Dyna-style Planning in High-dimensional State Spaces (2018, positive): “For instance Pan et al. (2018) and Goyal et al. (2018) use inverse dynamics models to effectively propagate value updates backwards in a manner similar to prioritized sweeping (Moore & Atke-son, 1993; Peng & Williams, 1993).”
Go-Explore: a New Approach for Hard-Exploration Problems (2019, background): “Contrary to Go-Explore, Recall Traces do not separate returning to states and exploring from those states, thus the algorithm helps ameliorate detachment, but not derailment.”
VIREL: A Variational Inference Framework for Reinforcement Learning (2018, methodology): “MERL is also compatible with methods that use recall traces [20].”
Hindsight Credit Assignment (2019, background): “[5] also use goal conditioning and learn a backtracking model, which predicts the state-action pairs occurring on trajectories that end up in goal states.”
InfoBot: Transfer and Exploration via the Information Bottleneck (2019, methodology): “9).”
Time Reversal as Self-Supervision (2018, background): “Another approach to self-supervised exploration involves resetting to goal states and exploring states around the goal state [35]–[37].”
Generative predecessor models for sample-efficient imitation learning (2019, background): “Predecessor models have a long history in reinforcement learning (e.g. Peng and Williams (1993)) with recent approaches using deep networks to generate off-policy transitions (Edwards et al., 2018; Pan et al., 2018) or to reinforce behavior leading to high-value states (Goyal et al., 2018).”
Imitating Latent Policies from Observation (2018, background): “Similar to recent works that predict state dynamics [7, 8], we learn the change in state ∆ t = s t +1 − s t , rather than the absolute next state, and compute s t +1 = s t + ∆ t .”
Competitive Experience Replay (2019, background): “Recently, Goyal et al. (2018) and Edwards et al. (2018) propose a learned backtracking model to generate traces that lead to high value states in order to obtain higher sample efﬁciency.”