Gated-Attention Architectures for Task-Oriented Language Grounding

By Devendra Singh Chaplot, Kanthashree Mysore Sathyendra, Rama Kumar Pasumarthi, Dheeraj Rajagopal, Ruslan Salakhutdinov. 2017

289 citations recorded by the source corpus.

To perform tasks specified by natural language instructions, autonomous agents need to extract semantically meaningful representations of language and map it to visual elements and actions in the environment. This problem is called task-oriented language grounding. We propose an end-to-end trainable neural architecture for task-oriented language grounding in 3D environments which assumes no prior linguistic or perceptual knowledge and requires only raw pixels from the environment and the natural language instruction as input. The proposed model combines the image and text representations using a Gated-Attention mechanism and learns a policy to execute the natural language instruction using standard reinforcement and imitation learning methods. We show the effectiveness of the proposed model on unseen instructions as well as unseen maps, both quantitatively and qualitatively. We also introduce a novel environment based on a 3D game engine to simulate the challenges of task-oriented language grounding over a rich set of instructions and environment states.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Model Architectures

Citations in the corpus

Environmental drivers of systematicity and generalization in a situated agent (2019, background): “Relative to more recent experiments on color-shape generalisation (Higgins et al., 2017; Chaplot et al., 2018; Yu et al., 2018), we study a wider range of phenomena, involving abstract modifying predicates (negation) and verbs (‘to lift’, ‘to put’) corresponding to complex behaviours requiring…”
Learning to Understand Goal Specifications by Modelling Reward (2018, background): “Recent work has shown deep reinforcement learning (RL) to be a promising paradigm for learning to follow language-like instructions in both 2D and 3D worlds (e.g. Hermann et al. (2017); Chaplot et al. (2018), see Section 4 for a review).”
Training Language Models with Language Feedback (2022, methodology): “Prior work uses language to specify the task (“instruction following” Chaplot et al., 2017; Mahmoudieh et al., 2022; Ouyang et al., 2022, inter alia ), drive exploration (Tam et al., 2022), infer reward functions (Lin et al., 2022; Sumers et al., 2021; Fidler et al., 2017, inter alia ), and train…”
LILA: Language-Informed Latent Actions (2021, background): “…sought to combine language and robotics, spanning approaches that map language to planning primitives [9, 30, 31, 32, 33], perform imitation learning from demonstrations and instructions [11, 34, 35, 36], and pair language instructions with reward functions for reinforcement learning [12, 37, 38].”
ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks (2019, background): “Platforms for translating language to action have become increasingly popular, spawning new test-beds [3, 12, 14, 41].”
Vision-and-Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments (2017, methodology): “This is typically achieved either by assuming that the set of all navigation goals, or objects to be acted upon, has been enumerated, and that each will be identified by label [11, 43], or by operating in severely restricted or simulated environments requiring limited perception [10, 16, 19, 24, 29,”
Embodied Question Answering (2017, background): “[16] both develop embodied agents in simple game-like environments to understand simple ‘go to X ’ or ‘pick up X ’ style commands where X could specify an object (and possibly some of its attributes).”
Hierarchical and Interpretable Skill Acquisition in Multi-task Reinforcement Learning (2017, background): “Recently, there has been work on grounding human language in 3D game environments (Hermann et al., 2017; Chaplot et al., 2017) or in text-based games (Narasimhan et al.”
IQA: Visual Question Answering in Interactive Environments (2017, methodology): “The learned baseline that we compare to (A3C), is based on a common architecture for reinforcement learning used in past works including for visual semantic planning [73] and task oriented language grounding [8, 19].”
Mastering the Dungeon: Grounded Language Learning by Mechanical Turker Descent (2017, neutral): “Grounding language in an interactive environment is an active area of research, however a number of recent works employ synthetic, templated language only (Sukhbaatar et al., 2015; Yu et al., 2017; Bordes et al., 2010; Hermann et al., 2017; Mikolov et al., 2015; Chaplot et al., 2017).”
Building Generalizable Agents with a Realistic and Rich 3D Environment (2018, methodology): “The idea of gated attention has been used in the past for language grounding (Chaplot et al., 2017), and transfer learning by language grounding (Narasimhan et al., 2017).”
A Survey of Reinforcement Learning Informed by Natural Language (2019, background): “More recently, with the developments in deep learning, a common approach has been to embed both the instruction and observation to condition the policy directly [Mei et al., 2016; Hermann et al., 2017; Chaplot et al., 2018; Janner et al., 2018; Misra et al., 2017; Chen et al., 2018].”
TextWorld: A Learning Environment for Text-based Games (2018, background): “Recently, multimodal visuo-linguistic environments were introduced to study grounded language learning through RL. Chaplot et al. [2017] customized the VizDoom platform [Kempka et al., 2016] for language grounding experiments: objects with certain properties are placed in a room and the agent is…”
BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning (2018, methodology): “First, BabyAI supports world state manipulation, missing in the visually appealing 3D environments of Hermann et al. (2017), Chaplot et al. (2018) and Wu et al. (2018).”
Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Text (2020, methodology): “Much recent work applies deep learning and policy optimization in end-to-end approaches to learning instruction following [Chaplot et al., 2018, Oh et al., 2017, Bahdanau et al., 2018, Chevalier-Boisvert et al., 2018, Yu et al., 2018b,a, Jiang et al., 2019].”
Learning to Navigate Unseen Environments: Back Translation with Environmental Dropout (2019, background): “…in the intersection of active perception and language understanding, several tasks have been proposed, including instruction-based navigation (Chaplot et al., 2018; Anderson et al., 2018b), target-driven navigation (Zhu et al., 2017b; Gupta et al., 2017), embodied question answering (Das et…”
TOUCHDOWN: Natural Language Navigation and Spatial Reasoning in Visual Street Environments (2018, background): “We ﬁnd that training with the PG-CB learning gives a lower shortest-path distance (SPD) while S UP gives better task completion (TC) and edit distance (ED) performance.”
Neural Modular Control for Embodied Question Answering (2018, background): “Recent deep reinforcement learning-based approaches to this explore it in 2D gridworlds [16, 20, 21], simple visual [22–27] and textual [28, 29] environments, perceptually-realistic 3D home simulators [1, 30–33], as well as real indoor scenes [34–36].”
Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration (2020, background): “This is inspired by a common goal-conditioned policy network architecture (Chaplot et al., 2018; Shu et al., 2017), which helps extract state information relevant to the goal.”