A Deep Hierarchical Approach to Lifelong Learning in Minecraft

By Chen Tessler, Shahar Givony, Tom Zahavy, Daniel Jaymin Mankowitz, Shie Mannor. 2016

404 citations recorded by the source corpus.

We propose a lifelong learning system that has the ability to reuse and transfer knowledge from one task to another while efficiently retaining the previously learned knowledge-base. Knowledge is transferred by learning reusable skills to solve tasks in Minecraft, a popular video game which is an unsolved and high-dimensional lifelong learning problem. These reusable skills, which we refer to as Deep Skill Networks, are then incorporated into our novel Hierarchical Deep Reinforcement Learning Network (H-DRLN) architecture using two techniques: (1) a deep skill array and (2) skill distillation, our novel variation of policy distillation (Rusu et. al. 2015) for learning skills. Skill distillation enables the H-DRLN to efficiently retain knowledge and therefore scale in lifelong learning, by accumulating knowledge and encapsulating multiple reusable skills into a single distilled network. The H-DRLN exhibits superior performance and lower learning sample complexity compared to the regular Deep Q Network (Mnih et. al. 2015) in sub-domains of Minecraft.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

Growing a Brain: Fine-Tuning by Increasing Model Capacity (2017, background): “In a broad sense, our approach is related to developmental learning [26, 16, 36] and lifelong learning [41, 25, 39, 29].”
Reinforcement Learning with Unsupervised Auxiliary Tasks (2016, neutral): “Labyrinth is comparable to other first-person 3D game platforms for AI research like VizDoom (Kempka et al., 2016) or Minecraft (Tessler et al., 2016).”
Progressive Neural Networks (2016, methodology): “2The Fisher of individual neurons (fully connected) and feature maps (convolutional layers) are computed over ρπ(k)(s, a).”
Embodied Question Answering (2017, methodology): “ntrol problem such that a higher-level planner invokes lowerlevel controls to issue primitive actions. Such hierarchical modeling has recently shown promise in the deep reinforcement learning setting [23,28,33]. Our model also draws inspiration from the work on Adaptive Computation Time models of Gr”
Continual Lifelong Learning with Neural Networks: A Review (2018, background): “licies from a simulated to a real environment. The task consists of learning pixel-to-action reinforcement learning policies with sparse rewards from raw visual input to a physical robot manipulator. Tessler et al. (2017) introduced a hierarchical deep reinforcement learning network that uses an arr”
A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “A distinctive feature of CRL is its high diversity of applications and benchmarks, including continuous control [203], [279], [296], [449], maze navigation [274], [370], video games (Atari [10], [59], [195], [220], [370], [429], StarCraft [372], Minecraft [405], Hanabi [312]), etc.”
Hierarchical and Interpretable Skill Acquisition in Multi-task Reinforcement Learning (2017, methodology): “…local policies that map states to primitive actions for achieving subgoals, and a global policy that initiates suitable subgoals in a sequence to achieve the ﬁnal goal of a task (Bacon & Precup, 2015; Kulkarni et al., 2016; Vezhnevets et al., 2016; Tessler et al., 2017; Andreas et al., 2017).”
Graying the black box: Understanding DQNs (2016, methodology): “The ability to automatically detect sub-goals and skills can also be used in the future as a component of an hierarchical DRL algorithm (Tessler et al., 2017; Kulkarni et al., 2016).”
Selective Experience Replay for Lifelong Learning (2018, background): “2016; Yin and Pan 2017) and discovering action hierarchies (Tessler et al. 2016), but these methods become inefficient when scaling to multiple tasks.”
CoMPS: Continual Meta Policy Search (2021, background): “Both terms describe the process of learning tasks in sequence while avoiding the problem of forgetting (French, 1999; Rusu et al., 2015; Parisotto et al., 2016; Rusu et al., 2016; Chen et al., 2015; Tessler et al., 2016; Aljundi et al., 2019).”
Continual World: A Robotic Benchmark For Continual Reinforcement Learning (2021, background): “[47] Chen Tessler, Shahar Givony, Tom Zahavy, Daniel J.”
Learning latent representations across multiple data domains using Lifelong VAEGAN (2020, background): “The current state-of-the-art deep learning approaches perform well on many individual databases [18,55], but suer from catastrophic forgetting when attempting to learn data associated with new tasks [3,13,19,51,62,65]. For example when a deep neural network is trained on a new database, its paramete”
FeUdal Networks for Hierarchical Reinforcement Learning (2017, background): “Recently (Tessler et al., 2016; Kulkarni et al., 2016) have demonstrated that combining deep learning with predefined sub-goals delivers promising results in challenging environments like Minecraft and Atari, however sub-goal discovery was not addressed.”
Learning to Navigate in Complex Environments (2016, methodology): “Learning to predict the binary loop label is done by minimizing the Bernoulli loss L l between l t and the output of a single-layer output from the hidden representation h t of the last hidden layer of the model, followed by a sigmoid activation.”
Active Long Term Memory Networks (2016, background): “The recent interest in bridging human and machine computations [5, 6, 7, 8] obliges to consider a learning framework that is continual, sequential in nature and potentially lifelong [9, 10].”
DeepMind Lab (2016, methodology): “Among 3D platforms for AI research, DeepMind Lab is comparable to others like VizDoom (Kempka et al., 2016) and Minecraft (Johnson et al., 2016; Tessler et al., 2016).”
Taskonomy: Disentangling Task Transfer Learning (2018, background): “Multi-task learning has experienced recent progress and the reported advantages are another support for existence of a useful structure among tasks [90, 97, 48, 73, 70, 48, 16, 94, 58, 9, 63].”
IQA: Visual Question Answering in Interactive Environments (2017, background): “More recent approaches include [37] who propose hierarchical-DQN with an intrinsically motivated RL algorithm, [66] who use HRL to create a lifelong learning system that has the ability to reuse and transfer knowledge from one task to another, and [51] who use HRL to enable zero shot task generaliza”