Playing Atari with Deep Reinforcement Learning
By Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, Martin A. Riedmiller. arXiv.org, 2013
13,852 citations recorded by the source corpus.
We present the first deep learning model to successfully learn control policies directly from high-dimensional sensory input using reinforcement learning. The model is a convolutional neural network, trained with a variant of Q-learning, whose input is raw pixels and whose output is a value function estimating future rewards. We apply our method to seven Atari 2600 games from the Arcade Learning Environment, with no adjustment of the architecture or learning algorithm. We find that it outperforms all previous approaches on six of the games and surpasses a human expert on three of them.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Management Science and Operations Research, Reinforcement Learning
View Paper: https://arxiv.org/abs/1312.5602
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Yoshua Bengio
- Mohit Bansal
- Pieter Abbeel
- Jianfeng Gao
- Quoc V. Le
- Ruslan Salakhutdinov
- Percy Liang
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Jitendra Malik
- Xiaogang Wang
- Ming-Hsuan Yang
- Andrew Zisserman
- Bernt Schiele
In Management Science and Operations Research
- Sergey Levine
- Pieter Abbeel
- Ofir Nachum
- Shimon Whiteson
- Lihong Li
- R´emi Munos
- Nan Jiang
- George Tucker
In Reinforcement Learning
- Sergey Levine
- Pieter Abbeel
- Chelsea Finn
- Shimon Whiteson
- Ofir Nachum
- Shixiang Shane Gu
- Yuxi Li
- George Tucker
Citations in the corpus
- Trust Region Policy Optimization (2015, methodology): “…which are made available through the Arcade Learning Environment (Bellemare et al., 2013) The images were preprocessed following the protocol in Mnih et al (2013), and the policy was represented by the convolutional neural network shown in Figure 3, with two convolutional layers with 16…”
- Continuous control with deep reinforcement learning (2015, methodology): “Our contribution in this paper is to provide modifications to DPG, inspired by the success of DQN, which allow it to use neural network function approximators to learn effectively in large state and action spaces online.”
- Deep Reinforcement Learning from Human Preferences (2017, methodology): “The second set of tasks we consider is a set of seven Atari games in the Arcade Learning Environment (Bellemare et al., 2013), the same games presented in Mnih et al., 2013.”
- A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark (2019, background): “Benchmarks have been critical in other sub-fields, such as RL (Mnih et al., 2013), image classification (Russakovsky et al., 2015), and NLP (Wang et al., 2018).”
- Deep Perm-Set Net: Learn to predict sets with unknown permutation and cardinality using deep neural networks (2018, background): “…(RNN) neural networks have enjoyed great success in many real-world problems, including scene classification Krizhevsky et al., semantic segmentation Papandreou et al. (2015), speech recognition Hinton et al. (2012), gaming Mnih et al. (2013; 2015), and image captioning Johnson et al. (2016).”
- Asynchronous Methods for Deep Reinforcement Learning (2016, methodology): “By storing the agent’s data in an experience replay memory, the data can be batched (Riedmiller, 2005; Schulman et al., 2015a) or randomly sampled (Mnih et al., 2013; 2015; Van Hasselt et al., 2015) from different time-steps.”
- End-to-End Training of Deep Visuomotor Policies (2015, methodology): “CNNs have also been trained to play video games with temporal difference learning and Monte Carlo tree search [9, 26].”
- From Pixels to Torques: Policy Learning with Deep Dynamical Models (2015, background): “M L ] 10 F e b 2015 of computer games, where human-level game strategies are learned autonomously, purely based on pixel information (Mnih et al., 2013).”
- Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog (2019, background): “Given a trajectory of actions, τ = {a1, a2, ...at−1}, let q(τ) = ∏T t=1 πθ(at, st) be the policy of our Q-learning algorithm at the trajectory level.”
- Reward learning from human preferences and demonstrations in Atari (2018, methodology): “We’ve selected 9 Atari games, 6 of which (Beamrider, Breakout, Enduro, Pong, Q*bert, and Seaquest) feature in Mnih et al. (2013) and Christiano et al. (2017).”
- Aligning Language Models with Preferences through f-divergence Minimization (2023, methodology): “The overall objective with the KL-penalty is maximized using an RL algorithm of choice including: PPO (Schulman et al., 2017) as in Ziegler et al. (2019) or Q-learning (Mnih et al., 2013) as in Jaques et al. (2017).”
- On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgetting (2022, methodology): “(2022) or Q-learning (Mnih et al., 2013) as in Jaques et al.”
- Human-centric Dialog Training via Offline Reinforcement Learning (2020, methodology): “Most similar to our work is Batch Constrained Q-learning (BCQ) (Fujimoto et al., 2018), which addresses extrapolation error in offline RL by constraining the actions of the policy to be close to the offline data.”
- Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-control (2016, methodology): “…(Mnih et al., 2013), a neural network called the deep Q-network (DQN) is trained to approximate Q ( s, a ; θ ) , using the following objective, where β is the exploration policy, and θ − is the parameters of the target Q-network (Mnih et al., 2013) that is held fixed during the gradient computation.”
- Pre-Trained Language Models for Interactive Decision-Making (2022, methodology): “Policy learning and LM. Traditional policy learning methods, such as PPO [37], DQN [27], DDPG [21], A3C [26], perform well on playing tasks on Atari, OpenAI gym [5], and MuJoCo [41].”
- Decision Transformer: Reinforcement Learning via Sequence Modeling (2021, methodology): “Some prior methods for reinforcement learning bear more resemblance to static supervised learning, such as Q-learning (Watkins, 1989; Mnih et al., 2013), which still uses iterative backups, or likelihood-based methods such as behavior cloning, which do not (discussed in previous section).”
- Measuring the Algorithmic Efficiency of Neural Networks (2020, background): “For instance, if we imagine simply scaling up a DQN [Mnih et al., 2013] model to play Go it could easily have needed 1000x or more times as much compute to reach AlphaGo level.”
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor (2018, background): “Model-free deep reinforcement learning (RL) algorithms have been applied in a range of challenging domains, from games (Mnih et al., 2013; Silver et al., 2016) to robotic control (Schulman et al., 2015).”