Tell me Dave: Context-sensitive grounding of natural language to manipulation instructions

By Dipendra Kumar Misra, Jaeyong Sung, Kevin Lee, Ashutosh Saxena. 2014

309 citations recorded by the source corpus.

It is important for a robot to be able to interpret natural language commands given by a human. In this paper, we consider performing a sequence of mobile manipulation tasks with instructions described in natural language. Given a new environment, even a simple task such as boiling water would be performed quite differently depending on the presence, location and state of the objects. We start by collecting a dataset of task descriptions in free-form natural language and the corresponding grounded task-logs of the tasks performed in an online robot simulator. We then build a library of verb–environment instructions that represents the possible instructions for each verb in that environment, these may or may not be valid for a different environment and task context. We present a model that takes into account the variations in natural language and ambiguities in grounding them to robotic instructions with appropriate environment context and task constraints. Our model also handles incomplete or noisy natural language instructions. It is based on an energy function that encodes such properties in a form isomorphic to a conditional random field. We evaluate our model on tasks given in a robotic simulator and show that it successfully outperforms the state of the art with 61.8% accuracy. We also demonstrate a grounded robotic instruction sequence on a PR2 robot using the Learning from Demonstration approach.

Subfields: Artificial Intelligence, Control and Systems Engineering, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (2022, methodology): “Machine learning has recently been applied to enable abstract task specification, allow general primitives, or relax constraints [67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78].”
Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents (2022, background): “A series of them parse language instructions into formal logic or rely mainly on lexical analysis to resolve various linguistic ambiguities for embodied agents [2, 28, 29, 44].”
ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks (2019, background): “In general, they are limited to a few scenes [34], consider a small number of objects [29], or use the same environment for training and testing [7].”
Gated-Attention Architectures for Task-Oriented Language Grounding (2017, background): “Artzi and Zettlemoyer (2013) focus on grounding natural language instructions to sequence of actions in a 2D environment, while Misra et al. (2016) addresses a similar challenge in a complex 3D environment.”
Mapping Instructions and Visual Observations to Actions with Reinforcement Learning (2017, background): “…including with semantic parsing (Chen and Mooney, 2011; Kim and Mooney, 2012, 2013; Artzi and Zettlemoyer, 2013; Artzi et al., 2014a,b; Misra et al., 2015, 2016), alignment models (Andreas and Klein, 2015), reinforcement learning (Branavan et al., 2009, 2010; Vogel and Jurafsky,…”
Self-Monitoring Navigation Agent via Auxiliary Progress Estimation (2019, background): “…an agent with the ability to interact with its surrounding environment for a particular goal such as object manipulation with instructions (Misra et al., 2016; Arkin et al., 2017), grounded language acquisition (Al-Omari et al., 2017; Kollar et al., 2013; Spranger & Steels, 2015; Dubba et…”
Reward Learning from Narrated Demonstrations (2018, background): “For instance, paired examples of instruction and action sequences have been collected through Amazon Mechanical Turk [8, 36, 37, 52].”
Few-shot Object Grounding and Mapping for Natural Language Robot Instruction Following (2020, background): “Natural language instruction following on physical robots is most commonly studied using handengineered symbolic representations of world state or instruction semantics [1, 7, 2, 3, 4, 8, 5, 9], which require representation design and state estimation pipelines that are hard to scale to complex envi”
Interactive Learning from Activity Description (2021, background): “…2015; Ling & Fidler, 2017; Goyal et al., 2019; Fu et al., 2019; Sumers et al., 2020), learning to ground language to actions (Chen & Mooney, 2011; Misra et al., 2014; Bisk et al., 2016; Liu et al., 2016; Wang et al., 2016; Li et al., 2017; 2020a;b), or devising EM-based algo-Interactive rithms…”
Grounding English Commands to Reward Functions (2015, methodology): “Unlike previous approaches that learn word groundings from pairs of natural language instructions and demonstrations of corresponding high-level actions [22, 9] or approaches that try to fill in the gaps when the learned actions do not match [18], our method learns mappings of natural language instr”
A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms (2019, background): “Conversely, knowledge of pre- and postconditions of skills can help to ground natural language commands to those skills, especially when the language description is incomplete [263].”
WebArena: A Realistic Web Environment for Building Autonomous Agents (2023, background): “As a result, the functionality of many environments is a limited version of their real-world counterparts, leading to a lack of task diversity within the environment [27, 1, 10, 23, 29, 30, 37].”
Learning Interpretable Spatial Operations in a Rich 3D Blocks World (2017, background): “2011), as acquiring a large corpus of natural language grounded with a real robot is prohibitively expensive (Misra et al. 2014; Thomason et al. 2017).”
Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation (2022, methodology): “Several works have proposed methods for grounding language in robot actions [49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60].”