Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning

By Junhyuk Oh, Satinder Singh, Honglak Lee, Pushmeet Kohli. 2017

288 citations recorded by the source corpus.

As a step towards developing zero-shot task generalization capabilities in reinforcement learning (RL), we introduce a new RL problem where the agent should learn to execute sequences of instructions after learning useful skills that solve subtasks. In this problem, we consider two types of generalizations: to previously unseen instructions and to longer sequences of instructions. For generalization over unseen instructions, we propose a new objective which encourages learning correspondences between similar subtasks by making analogies. For generalization over sequential instructions, we present a hierarchical architecture where a meta controller learns to use the acquired skills for executing the instructions. To deal with delayed reward, we propose a new neural architecture in the meta controller that learns when to update the subtask, which makes learning more efficient. Experimental results on a stochastic 3D domain show that the proposed ideas are crucial for generalization to longer instructions as well as unseen instructions.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

Learning with Latent Language (2017, methodology): “Structured language-like annotations have been used to improve learning of generalizable structured policies (Oh et al., 2017; Andreas et al., 2017; Denil et al., 2017).”
Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (2022, background): “Most of these prior works focus on following low-level instructions, such as for pick-and-place tasks and other robotic manipulation primitives [20, 22, 56, 21, 13, 26], though some methods address long-horizon, compound tasks in simulated domains [57, 58, 54].”
FiLM: Visual Reasoning with a General Conditioning Layer (2017, methodology): “However, approaches from word embeddings, representation learning, and zero-shot learning can be applied to directly optimize ( γ , β ) for analogy-making (Bordes et al. 2013; Guu, Miller, and Liang 2015; Oh et al. 2017).”
Embodied Question Answering (2017, background): “Hierarchical modeling has recently shown promise in deep reinforcement learning [22,28,36].”
Continual and Multi-Task Architecture Search (2019, positive): “…is primarily used to improve the generalization performance of a task by leveraging knowledge from related tasks (Caru-ana, 1998; Collobert and Weston, 2008; Girshick, 2015; Luong et al., 2015; Ruder et al., 2017; Au-genstein et al., 2018; Guo et al., 2018; Oh et al., 2017; Ruder and Plank, 2017).”
The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision (2019, background): “The integration of formal semantics into the processing of complex natural language would be meaningful future work (Artzi & Zettlemoyer, 2013; Oh et al., 2017).”
Learning to Speak and Act in a Fantasy Text Adventure Game (2019, background): “2016), but they typically are suited to reinforcement learning of actions, and involve templated language for navigation or question answering tasks, if at all (Oh et al., 2017; Yi et al., 2018).”
Automatically Composing Representation Transformations as a Means for Generalization (2018, background): “Our work is one of a small but growing community [32, 40, 61] linking metareasoning with deep learning.”
Building Generalizable Agents with a Realistic and Rich 3D Environment (2018, background): “Generalization: There is a recent trend in reinforcement learning focusing on the problem of generalization, ranging from learning to plan (Tamar et al., 2016), meta-learning (Duan et al., 2016; Finn et al., 2017a) to zero-shot learning (Andreas et al., 2016; Oh et al., 2017; Higgins et al., 2017).”
A Survey of Reinforcement Learning Informed by Natural Language (2019, methodology): “For example, [Oh et al. , 2017]”
Good-Enough Compositional Data Augmentation (2019, methodology): “There are many sophisticated tools available for improving the function approximators or loss functions themselves—regularization (Oh et al., 2017), posterior regularization (Ganchev et al., 2010; Hu et al., 2018), explicit stacks (Grefenstette et al., 2015) and composition operators (Bowman et…”
Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Text (2020, methodology): “Much recent work applies deep learning and policy optimization in end-to-end approaches to learning instruction following [Chaplot et al., 2018, Oh et al., 2017, Bahdanau et al., 2018, Chevalier-Boisvert et al., 2018, Yu et al., 2018b,a, Jiang et al., 2019].”
Ray Interference: a Source of Plateaus in Deep Reinforcement Learning (2019, background): “Our investigation has natural connections to the ﬁeld of multi-task learning (be it for supervised or RL tasks [25, 31]), namely by considering that the multitask objective is additive over one J k per task.”
Language Conditioned Imitation Learning Over Unstructured Data (2020, background): “However, prior work has often studied 2D environments [24, 32, 27, 52] and simpliﬁed actuators [48, 18, 12].”
Hindsight policy gradients (2017, neutral): “Recent examples of this approach include the works of Oh et al. [31], Vezhnevets et al. [32], Kulkarni et al. [33].”