Playing FPS Games with Deep Reinforcement Learning

By Guillaume Lample, Devendra Singh Chaplot. 2016

635 citations recorded by the source corpus.

Advances in deep reinforcement learning have allowed autonomous agents to perform well on Atari games, often outperforming humans, using only raw pixels to make their decisions. However, most of these games take place in 2D environments that are fully observable to the agent. In this paper, we present the first architecture to tackle 3D environments in first-person shooter games, that involve partially observable states. Typically, deep reinforcement learning methods only utilize visual input for training. We present a method to augment these models to exploit game feature information such as the presence of enemies or items, during the training phase. Our model is trained to simultaneously learn these features along with minimizing a Q-learning objective, which is shown to dramatically improve the training speed and performance of our agent. Our architecture is also modularized to allow different models to be independently trained for different phases of the game. We show that the proposed architecture substantially outperforms built-in AI agents of the game as well as average humans in deathmatch scenarios.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

Off-Policy Actor-Critic with Shared Experience Replay (2019, background): “Finally, it is often useful to learn multiple general value functions (Sutton et al., 2011; Mankowitz et al., 2018; Lample & Chaplot, 2016; Mirowski et al., 2017; Jaderberg et al., 2017b) or options (Sutton et al., 1999; Bacon et al., 2017) from a single stream of experience.”
Shaping Belief States with Generative Environment Models for RL (2019, background): “The challenge of model-based RL in rich 3D environments has compelled some researchers to use privileged information such as camera-locations [15], depth information [53], and other ground-truth state variables of the state simulator [54, 49].”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “A common approach to enable learning about more than a specific task is to augment the loss function with auxiliary losses to provide denser training signals in RL (Li, Li, Gao, He, Chen, Deng, and He, 2015; Lample and Chaplot, 2017).”
Learning to Act by Predicting the Future (2016, background): “That submission, described by Lample & Chaplot (2016), constitutes a strong baseline.”
Factorized Q-learning for large-scale multi-agent systems (2018, methodology): “RIAL (Reinforced Inter-Agent Learning) [3] combines the idea of IQL with DRQN [7] to learn communication protocols in a cooperative multi-agent environment.”
Lenient Multi-Agent Deep Reinforcement Learning (2017, background): “Deep reinforcement learning agents have been shown to master numerous complex problem domains, ranging from computer games [15, 19, 25, 33] to robotics tasks [8, 10].”
Investigating Recurrence and Eligibility Traces in Deep Q-Networks (2017, positive): “Like in Lample & Singh Chaplot (2016), we therefore sample a slightly longer length of trajectory and use the ﬁrst m states to ﬁll the hidden state.”
Learning to Navigate in Complex Environments (2016, background): “…model on the hidden representations to predict the next observed state, in the context of imitation learning of sequences provided by experts, and Lample & Chaplot (2016) show that the performance of a DQN agent in a ﬁrst-person shooter game in the VizDoom environment can be substantially…”
Multi-task Deep Reinforcement Learning with PopArt (2018, background): “Several agents (Jaderberg et al. 2016; Lample and Chaplot 2016; Shelhamer et al. 2016; Mirowski et al. 2016) demonstrated this by improving performance on a single external task by learning off-policy about auxiliary tasks deﬁned on the same stream of experience (e.g. pixel control, immediate…”
Learning Attentional Communication for Multi-Agent Cooperation (2018, background): “Deep reinforcement learning (RL) has achieved remarkable success in a series of challenging problems, such as game playing [16][22][8] and robotics [12][11][5].”
Gated-Attention Architectures for Task-Oriented Language Grounding (2017, methodology): “Prior work has explored using Deep Re-inforcement learning approaches for playing FPS games (Lample and Chaplot, 2016; Kempka et al., 2016; Kulkarni et al., 2016).”
On Evaluation of Embodied Navigation Agents (2018, background): “A small departure from purely reactive architectures is to equip the agent with short-term vectorial memory maintained by a recurrent update mechanism [14, 17, 20].”
Human-level performance in 3D multiplayer games with population-based reinforcement learning (2018, background): “Previous work on ﬁrst-person games considers either much simpler games ( 30, 36, 45, 75 ), simpliﬁed agent interfaces ( 68 ), or non-learning systems ( 50, 69 ), and previously studied multi-agent domains often consist of discrete-state environments ( 18, 38, 64 ), have simpliﬁed 2D dynamics ( 23,…”
Learning by Playing - Solving Sparse Reward Tasks from Scratch (2018, background): “…connected to learning to predict the future via “successor” representations (Dayan, 1993; Kulka-rni et al., 2016b; Barreto et al., 2017) or forecasts (Schaul & Ring, 2013; Lample & Chaplot, 2017; Dosovitskiy & Koltun, 2017) and are trained to be predictive of features extracted from future states.”
Deep Variational Reinforcement Learning for POMDPs (2018, background): “Unlike the losses proposed by Lample & Chaplot (2017), we do not require additional information from the environment.”
PRIMAL: Pathfinding via Reinforcement and Imitation Multi-Agent Learning (2018, methodology): “We also added a feature layer connected to the LSTM [32], to train”
Learning macromanagement in starcraft from replays using deep learning (2017, neutral): “Combined with reinforcement learning, these techniques have surpassed the previous state-of-the-art in playing Atari games [16], the classic board game Go [23] and the 3D ﬁrst-person shooter Doom [15].”
Active Neural Localization (2018, methodology): “This technique of domain randomization has shown to be effective in generalizing to unknown maps within the simulation environment (Lample & Chaplot, 2016) and transferring from simulation to real-world (Tobin et al., 2017).”