Learning to Play in a Day: Faster Deep Reinforcement Learning by Optimality Tightening

By Frank S. He, Yang Liu, Alexander Gerhard Schwing, Jian Peng. 2016

87 citations recorded by the source corpus.

We propose a novel training algorithm for reinforcement learning which combines the strength of deep Q-learning with a constrained optimization approach to tighten optimality and encourage faster reward propagation. Our novel technique makes deep reinforcement learning more practical by drastically reducing the training time. We evaluate the performance of our approach on the 49 games of the challenging Arcade Learning Environment, and report significant improvements in both training time and accuracy.

Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Addressing Function Approximation Error in Actor-Critic Methods (2018, methodology): “These methods have been shown to be an effective approach, through importance sampling (Precup et al., 2001; Munos et al., 2016), distributed methods (Mnih et al., 2016; Espeholt et al., 2018), and approximate bounds (He et al., 2016).”
Rainbow: Combining Improvements in Deep Reinforcement Learning (2017, methodology): “Optimality tightening (He et al. 2016) uses multi-step returns to construct additional inequality bounds, instead of just replacing the 1-step targets in Qlearning.”
Self-Imitation Learning (2018, neutral): “Optimality tightening (He et al., 2017) introduced an objective based on the lower/upper bound of the optimal Qfunction, which is similar to a part of our theoretical result.”
Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research (2018, background): “Another approach would be to use multi-step optimality tightening inequalities (He et al., 2016).”
Neural Episodic Control (2017, methodology): “Several authors have proposed methods of improving reward propagation and the back up mechanism of Q learning (Harutyunyan et al., 2016; Munos et al., 2016; He et al., 2016) by incorporating on-policy rewards or by adding constraints to the optimisation.”
Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learning (2016, background): “Averaged-DQN is a simple extension that can be easily integrated with other DQN variants such as Schaul et al. (2015); Van Hasselt et al. (2015); Wang et al. (2015); Bellemare et al. (2016); He et al. (2016).”
Learning What Data to Learn (2017, background): “As to future work, our ﬁrst goal is to provide more efﬁcient way for NDF training, for example by optimality tightening (He et al., 2017), by Actor-Critic to collect more frequent reward signals 6 , or by learning from pure scratch to eliminate the feed-forward step in current design to obtain the…”
Sample-Efficient Deep Reinforcement Learning via Episodic Backward Update (2018, methodology): “Furthermore, our method requires only about 37% of computation time used by Optimality Tightening.”
Regularize, Expand and Compress: NonExpansive Continual Learning (2020, methodology): “The AutoML training time is another limitation with REC, however it can be further improved by optimality tightening [9] or parallelization [34] or similar approaches for reducing the training time.”
The Reactor: A fast and sample-efficient Actor-Critic agent for Reinforcement Learning (2017, background): “e DQN framework, act -greedily according to an action-value function and train using minibatches of transitions sampled from an experience replay buffer (Van Hasselt et al., 2016; Wang et al., 2015; He et al., 2017; Anschel et al., 2017). These value-function agents beneﬁt from improved sample compl”
Deep Reinforcement Learning: An Overview (2017, methodology): “…uses raw inputs without manual feature engineering to generate outputs, e.g., AlexNet (Krizhevsky et al., 2012) with raw pixels for image classiﬁcation, Seq2Seq (Sutskever et al., 2014) with raw sentences for machine translation, and DQN (Mnih et al., 2015) with raw pixels and score to play games.”