ViZDoom: A Doom-based AI research platform for visual reinforcement learning

By Michal Kempka, Marek Wydmuch, Grzegorz Runc, Jakub Toczek, Wojciech Jaśkowski. 2016

752 citations recorded by the source corpus.

The recent advances in deep neural networks have led to effective vision-based reinforcement learning methods that have been employed to obtain human-level controllers in Atari 2600 games from pixel data. Atari 2600 games, however, do not resemble real-world tasks since they involve non-realistic 2D environments and the third-person perspective. Here, we propose a novel test-bed platform for reinforcement learning research from raw visual information which employs the first-person perspective in a semi-realistic 3D world. The software, called ViZDoom, is based on the classical first-person shooter video game, Doom. It allows developing bots that play the game using the screen buffer. ViZDoom is lightweight, fast, and highly customizable via a convenient mechanism of user scenarios. In the experimental part, we test the environment by trying to learn bots for two scenarios: a basic move-and-shoot task and a more complex maze-navigation problem. Using convolutional deep neural networks with Q-learning and experience replay, for both scenarios, we were able to train competent bots, which exhibit human-like behaviors. The results confirm the utility of ViZDoom as an AI research platform and imply that visual reinforcement learning in 3D realistic first-person perspective environments is feasible.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

RL$^2$: Fast Reinforcement Learning via Slow Reinforcement Learning (2016, methodology): “The visual navigation experiment in Section 3.3, is actually an instance of the this POMDP setting.”
VideoGPT: Video Generation using VQ-VAE and Transformers (2021, methodology): “We present qualitative results on the BAIR Robot Pushing dataset and Vizdoom simulator (Kempka et al., 2016).”
OpenAI Gym (2016, methodology): “Since the initial release, more environments have been created, including ones based on the open source physics engine Box2D or the Doom game engine via VizDoom [15].”
Vision-and-Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments (2017, methodology): “Navigation based simulator Our simulator is similar in spirit to recently proposed 3D RL environments based on game engines, such as ViZDoom [22], DeepMind Lab [7] and THOR [51].”
Recurrent World Models Facilitate Policy Evolution (2018, methodology): “In this experiment, we train an agent inside the environment generated by its world model trained to mimic a VizDoom [40] environment.”
Reinforcement Learning with Unsupervised Auxiliary Tasks (2016, methodology): “Unlike in VizDoom, agents can look up to the sky or down to the ground.”
Hierarchical and Interpretable Skill Acquisition in Multi-task Reinforcement Learning (2017, background): “Deep reinforcement learning has demonstrated success in policy search for tasks in domains like game playing (Mnih et al., 2015; Silver et al., 2016; 2017; Kempka et al., 2016; Mirowski et al., 2017) and robotic control (Levine et al.”
Curiosity-Driven Exploration by Self-Supervised Prediction (2017, background): “Environments The first environment we evaluate on is the VizDoom (Kempka et al., 2016) game.”
A Deep Hierarchical Approach to Lifelong Learning in Minecraft (2016, background): “This architecture also has the potential to be utilized in other 3D domains such as Doom (Kempka et al. 2016) and Labyrinth (Mnih et al.”
Continuous Coordination As a Realistic Scenario for Lifelong Learning (2021, methodology): “(2020) designed CRLMaze based on VizDoom (Kempka et al., 2016), an object-picking LLL task that is composed of 4 scenarios (Light, Texture, Object, All) of incremental difficulty and a total of 12 maps.”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “For more complex agents that can play long-horizon, strategy games accompanied with rich visual observations, several algorithms have also explored Minecraft (Duncan, 2011) and VizDoom (Kempka, Wydmuch, Runc, Toczek, and Jaśkowski, 2016).”
Continual Reinforcement Learning in 3D Non-stationary Environments (2019, methodology): “VizDoom is a reinforcement learning API build around the famous ZDoom game engine and providing all the necessary utilities to train a RL agent in arbitrary complex environments.”
Learning to Act by Predicting the Future (2016, methodology): “To interface with the game engine, we use the ViZDoom platform developed by Kempka et al. (2016).”
Reinforcement Learning from Imperfect Demonstrations (2018, background): “, 2016) and intelligent agents in Starcraft II and vizDoom (Vinyals et al., 2017; Kempka et al., 2016).”
TorchCraft: a Library for Machine Learning Research on Real-Time Strategy Games (2016, methodology): “Thus, we advocate here to have not only the pixels as input and keyboard/mouse for commands, as in [3, 4, 12], but also a structured representation of the game state, as in”
StarCraft II: A New Challenge for Reinforcement Learning (2017, methodology): “The ALE is a prominent example in a rich tradition of video game benchmarks for AI [30], including Super Mario [35], Ms Pac-Man [26], Doom [13], Unreal Tournament [11], as well as general video game-playing frameworks [29, 5] and competitions [23].”
AI2-THOR: An Interactive 3D Environment for Visual AI (2017, background): “DeepMind Lab [3] X X VizDoom [13] X X MINOS (Matterport3D) [17] X X MINOS (SUNCG) [17] X X X House3D [2] X X X HoME [5] X X X X AI2-THOR X X X X X Table 1.”
Target-driven visual navigation in indoor scenes using deep reinforcement learning (2016, methodology): “Similar frameworks have been proposed by [39], [40], [44], [42], [41], but the”
DeepMind Lab (2016, methodology): “Compared to VizDoom, DeepMind Lab is more removed from its origin in a first-person shooter genre video game.”