Control of Memory, Active Perception, and Action in Minecraft

By Junhyuk Oh, Valliappa Chockalingam, Satinder Singh, Honglak Lee. 2016

320 citations recorded by the source corpus.

In this paper, we introduce a new set of reinforcement learning (RL) tasks in Minecraft (a flexible 3D world). We then use these tasks to systematically compare and contrast existing deep reinforcement learning (DRL) architectures with our new memory-based DRL architectures. These tasks are designed to emphasize, in a controllable manner, issues that pose challenges for RL methods including partial observability (due to first-person visual observations), delayed rewards, high-dimensional visual observations, and the need to use active perception in a correct manner so as to perform well in the tasks. While these tasks are conceptually simple to describe, by virtue of having all of these challenges simultaneously they are difficult for current DRL architectures. Additionally, we evaluate the generalization performance of the architectures on environments not used during training. The experimental results show that our new architectures generalize to unseen environments better than existing DRL architectures.

Subfields: Artificial Intelligence, Biomedical Engineering, Computational Theory and Mathematics, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

In Computational Theory and Mathematics

In Reinforcement Learning

Citations in the corpus

RL$^2$: Fast Reinforcement Learning via Slow Reinforcement Learning (2016, background): “2016) have studied similar vision-based navigation tasks in Minecraft.”
Sample Efficient Actor-Critic with Experience Replay (2016, background): “…simulated environments, where agents can be trained to learn a large repertoire of cognitive skills, are at the core of recent breakthroughs in AI (Bellemare et al., 2013; Mnih et al., 2015; Schulman et al., 2015a; Narasimhan et al., 2015; Mnih et al., 2016; Brockman et al., 2016; Oh et al., 2016).”
Value Prediction Network (2017, background): “A lot of variations of DQN have been proposed for learning better state representation [37, 16, 9, 22, 36, 24], including the use of memory-based networks for handling partial observability [9, 22, 24], estimating both state-values and advantage-values as a decomposition of Q-values [37], learning s”
Reinforcement Learning with Unsupervised Auxiliary Tasks (2016, background): “Labyrinth also supports continuous motion unlike the Minecraft platform of (Oh et al., 2016), which is a 3D grid world.”
Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning (2017, background): “learning [13], [2], [9], actor-critic methods [14], [15], [16], and policy gradients [3], [17] have been shown to learn very complex skills in high-dimensional state spaces, including simulated robotic locomotion, driving, video game playing, and navigation.”
A Deep Hierarchical Approach to Lifelong Learning in Minecraft (2016, methodology): “Minecraft is an open research problem as it is impossible to solve the entire game using a single AI technique (Smith and Aha ; Oh et al. 2016).”
Learning to Act by Predicting the Future (2016, methodology): “Recent work has explored memory-based models (Oh et al., 2016) and integrating such ideas with the presented approach may yield substantial advances.”
Episodic Exploration for Deep Deterministic Policies: An Application to StarCraft Micromanagement Tasks (2016, background): “Both the work on Atari games [19] and the recent Minecraft scenarios studied by researchers [1, 22] focus on the control of a single agent, with a ﬁxed, limited set of actions.”
Actor-Attention-Critic for Multi-Agent Reinforcement Learning (2018, background): “Attention The attention mechanism functions in a manner similar to a differentiable key-value memory model (Graves et al., 2014; Oh et al., 2016).”
Learning modular neural network policies for multi-task and multi-robot transfer (2016, background): “Other recent work on high-dimensional neural network policy search has studied continuous control tasks for simulated robots [2], [25], playing Atari games [1], and other tasks [26].”
Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning (2017, methodology): “We developed a 3D visual environment using Minecraft based on Oh et al. (2016) where the agent can interact with many objects.”
Learning Robust Rewards with Adversarial Inverse Reinforcement Learning (2017, background): “d value functions, and has shown promising results on a range of complex tasks, from vision-based robotic control (Levine et al., 2016) to video games such as Atari (Mnih et al., 2015) and Minecraft (Oh et al., 2016). However, reward engineering remains a signiﬁcant barrier to applying reinforcement”
A Brief Survey of Deep Reinforcement Learning (2017, neutral): “A more common approach in deep learning is to utilise recurrent neural networks (RNNs) [138, 35, 36, 72, 82], which, unlike feedforward neural”
On Evaluation of Embodied Navigation Agents (2018, background): “The next level of architectural sophistication is to use more advanced memory mechanisms that support the construction of rich internal representations of the agent’s environment [12, 13, 22, 23, 27].”
Semi-parametric Topological Memory for Navigation (2018, methodology): “t been demonstrated to perform efﬁcient goal-directed navigation in previously unseen environments, and empirical results indicate that LSTM-based systems are not up to the task (Savva et al., 2017). Oh et al. (2016) use an addressable memory for ﬁrst-person-view navigation in three-dimensional envi”
Neural Episodic Control (2017, methodology): “We did not directly compare to DRQN (Hausknecht & Stone, 2015) nor FRMQN (Oh et al., 2016) as results were not available for all Atari games.”