Mapping Instructions and Visual Observations to Actions with Reinforcement Learning

By Dipendra Kumar Misra, J. Langford, Yoav Artzi. 2017

256 citations recorded by the source corpus.

We propose to directly map raw visual observations and text input to actions for instruction execution. While existing approaches assume access to structured environment representations or use a pipeline of separately trained models, we learn a single model to jointly reason about linguistic and visual input. We use reinforcement learning in a contextual bandit setting to train a neural network agent. To guide the agent’s exploration, we use reward shaping with different forms of supervision. Our approach does not require intermediate representations, planning procedures, or training different models. We evaluate in a simulated environment, and show significant improvements over supervised learning and common reinforcement learning variants.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

Learning to Understand Goal Specifications by Modelling Reward (2018, background): “2017) train RL agents to produce goal-states given instructions.”
Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (2022, background): “A large number of prior works have learned language-conditioned behavior via imitation learning [51, 22, 20, 13, 26, 37] or reinforcement learning [52, 53, 49, 54, 55, 56, 21, 41].”
ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks (2019, background): “There are several existing benchmarks based on simple block worlds and fully observable scenes [9, 33].”
Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-Training (2020, background): “While a number of methods [20, 21, 33] have been proposed to improve language understanding, common to all existing work is that the agent learns to understand each instruction from scratch or in isolation, without collectively leveraging prior visiongrounded domain knowledge.”
Vision-and-Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments (2017, background): “[35] introduce more complex language instructions, while keeping the environment (a virtual Blocks world) fully-observable and visually limited.”
Lifelong and Continual Learning Dialogue Systems: Learning during Conversation (2021, background): “For example, authors of (Vogel and Jurafsky 2010; Misra, Langford, and Artzi 2017; Tellex et al. 2020) have explored deep learning and reinforcement learning to ground NL commands directly into executable actions.”
Speaker-Follower Models for Vision-and-Language Navigation (2018, background): “While a number of approaches [32, 33, 53] have been proposed for the various navigation benchmarks, they generally employ a single model that learns to map directly from instructions to actions from a limited corpus of annotated trajectories.”
A Survey of Reinforcement Learning Informed by Natural Language (2019, methodology): “Human-generated natu-ral language instructions are used in [ MacMahon et al. , 2006; Bisk et al. , 2016; Misra et al. , 2017; Janner et al. , 2018; Chen et al. , 2018; Anderson et al. , 2018; Goyal et al. , 2019; Wang et al. , 2019 ] .”
Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Text (2020, background): “Vision and language navigation (VLN) models learn to follow natural language directions that are longer that those considered in this work Misra et al. [2017, 2018], Anderson et al. [2018], Fried et al. [2018], Wang et al. [2019], Zhu et al. [2019].”
Skill Induction and Planning with Latent Language (2021, background): “…at the intersection of language and control include instruction following (Chen and Mooney, 2011; Branavan et al., 2009; Tellex et al., 2011; Anderson et al., 2018; Misra et al., 2017), embodied question answering (Das et al., 2018; Gordon et al., 2018) and dialog tasks (Tellex et al., 2020).”
Learning to Navigate Unseen Environments: Back Translation with Environmental Dropout (2019, background): “This task has been studied by many works (Tellex et al., 2011; Chen and Mooney, 2011; Artzi and Zettlemoyer, 2013; Andreas and Klein, 2015; Mei et al., 2016; Misra et al., 2017) in recent years.”
TOUCHDOWN: Natural Language Navigation and Spatial Reasoning in Visual Street Environments (2018, methodology): “We use a shaped reward function R ( s, a, s (cid:48) ) = R p ( s, a, s (cid:48) ) + F ( s, a, s (cid:48) ) , where s is the source state, s (cid:48) is the target state, and a is the action.”
Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigation (2018, background): “[5] formulates navigation as a sequential-decision process and proposes to use reward shaping to effectively train the RL agent.”
Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learning (2019, background): “…in these problems remains simplistic (Table 1): they either do not incorporate the ability to request additional help while executing tasks (Misra et al., 2014, 2017; Anderson et al., 2018b; Chen et al., 2019; Das et al., 2018; Misra et al., 2018; Wijmans et al., 2019; Qi et al., 2019), or…”
Language Conditioned Imitation Learning Over Unstructured Data (2020, background): “However, prior work has often studied 2D environments [24, 32, 27, 52] and simpliﬁed actuators [48, 18, 12].”
Neural Modular Control for Embodied Question Answering (2018, background): “Recent deep reinforcement learning-based approaches to this explore it in 2D gridworlds [16, 20, 21], simple visual [22–27] and textual [28, 29] environments, perceptuallyrealistic 3D home simulators [1, 30–33], as well as real indoor scenes [34–36].”
Self-Monitoring Navigation Agent via Auxiliary Progress Estimation (2019, background): “Extensive efforts have been made to ground language instructions into a sequence of actions (MacMahon et al., 2006; Branavan et al., 2009; Vogel & Jurafsky, 2010; Tellex et al., 2011; Artzi & Zettlemoyer, 2013; Andreas & Klein, 2015; Mei et al., 2016; Cohn et al., 2016; Misra et al., 2017).”
Instruction-driven history-aware policies for robotic manipulations (2022, background): “Instruction-driven vision-based robotic manipulation has received growing attention for manipulations in 2D planar [57, 58] or recent 3D environments [8, 59, 60], and has been transferred to the real world [9, 10].”