Sample-Efficient Deep Reinforcement Learning via Episodic Backward Update
By Su Young Lee, Sung-Ik Choi, Sae-Young Chung. 2018
83 citations recorded by the source corpus.
We propose Episodic Backward Update (EBU) - a novel deep reinforcement learning algorithm with a direct value propagation. In contrast to the conventional use of the experience replay with uniform random sampling, our agent samples a whole episode and successively propagates the value of a state to its previous states. Our computationally efficient recursive algorithm allows sparse and delayed rewards to propagate directly through all transitions of the sampled episode. We theoretically prove the convergence of the EBU method and experimentally demonstrate its performance in both deterministic and stochastic environments. Especially in 49 games of Atari 2600 domain, EBU achieves the same mean and median human normalized performance of DQN by using only 5% and 10% of samples, respectively.
Subfields: Artificial Intelligence, Management Science and Operations Research
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Yoshua Bengio
- Pieter Abbeel
- Quoc V. Le
- J. Urgen Schmidhuber
- Peter Stone
- Ruslan Salakhutdinov
- Jianfeng Gao
In Management Science and Operations Research
- Yoshua Bengio
- Mark Rowland
- Matthew E. Taylor
- William Fedus
- Hugo Larochelle
- José Antonio Arjona-Medina
- Michael Widrich
- Peter Stone
Citations in the corpus
- RUDDER: Return Decomposition for Delayed Rewards (2018, methodology): “The lessons replay buffer is an episodic memory, which has been used for episodic control [64] and for episodic backward update to efficiently propagate delayed rewards [63].”
- Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey (2020, methodology): “One example is the episodic backward update (EBU) method developed by Lee et al. (2019). In order to speed up the propagation of delayed rewards (e.”
- Revisiting Fundamentals of Experience Replay (2020, background): “Beyond direct manipulation of these properties of the buffer, improving and understanding experience replay algorithms remains an active area of research (Pan et al., 2018; Schlegel et al., 2019; Zha et al., 2019; Novati & Koumoutsakos, 2019; Sun et al., 2020; Lee et al., 2019).”