Forward-Backward Reinforcement Learning

By Ashley D. Edwards, Laura Downs, James C. Davidson. 2018

44 citations recorded by the source corpus.

Goals for reinforcement learning problems are typically defined through hand-specified rewards. To design such problems, developers of learning algorithms must inherently be aware of what the task goals are, yet we often require agents to discover them on their own without any supervision beyond these sparse rewards. While much of the power of reinforcement learning derives from the concept that agents can learn with little guidance, this requirement greatly burdens the training process. If we relax this one restriction and endow the agent with knowledge of the reward function, and in particular of the goal, we can leverage backwards induction to accelerate training. To achieve this, we propose training a model to learn to take imagined reversal steps from known goal states. Rather than training an agent exclusively to determine how to reach a goal while moving forwards in time, our approach travels backwards to jointly predict how we got there. We evaluate our work in Gridworld and Towers of Hanoi and empirically demonstrate that it yields better performance than standard DDQN.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

RUDDER: Return Decomposition for Delayed Rewards (2018, methodology): “ies increases exponentially with the delay of the reward. Instead of using a forward view, we propose to adopt a backward view approach. A backward view approach can trace back from known goal states [17] or from high reward states [29]. A recent approach to backward view uses attention-based method”
Competitive Experience Replay (2019, background): “Recently, Goyal et al. (2018) and Edwards et al. (2018) propose a learned backtracking model to generate traces that lead to high value states in order to obtain higher sample efﬁciency.”
Imitating Latent Policies from Observation (2018, result): “Similar to recent works that predict state dynamics (Edwards et al., 2018; Goyal et al., 2018), our approach predicts the differences between states ∆t = st+1 − st, rather than the absolute next state, and computes st+1 = st + ∆t.”
Time Reversal as Self-Supervision (2018, background): “Another approach to self-supervised exploration involves resetting to goal states and exploring states around the goal state [35]–[37].”
Backplay: "Man muss immer umkehren" (2018, unknown): “Goyal et al. (2018) and Edwards et al. (2018) simultaneously proposed the use of a learned backtracking model to generate traces that lead to high value states.”
Learning Compositional Neural Programs with Recursive Tree Search and Planning (2019, background): “In [Edwards et al., 2018], the authors introduced a backward induction, to enable the agent to reason backwards in time.”
Generative predecessor models for sample-efficient imitation learning (2019, background): “Predecessor models have a long history in reinforcement learning (e.g. Peng and Williams (1993)) with recent approaches using deep networks to generate off-policy transitions (Edwards et al., 2018; Pan et al., 2018) or to reinforce behavior leading to high-value states (Goyal et al., 2018).”
Recall Traces: Backtracking Models for Efficient Reinforcement Learning (2018, background): “Finally, (Edwards et al., 2018) is concurrent work to this and also proposes training on imagined reversal steps from known goal states.”