Natural Language Communication with Robots

By Yonatan Bisk, Deniz Yuret, Daniel Marcu. 2016

127 citations recorded by the source corpus.

We propose a framework for devising empirically testable algorithms for bridging the communication gap between humans and robots. We instantiate our framework in the context of a problem setting in which humans give instructions to robots using unrestricted natu-ral language commands, with instruction sequences being subservient to building complex goal conﬁgurations in a blocks world. We show how one can collect meaningful training data and we propose three neural architectures for interpreting contextually grounded natural language commands. The proposed architectures allow us to correctly understand/ground the blocks that the robot should move when instructed by a human who uses unrestricted language. The architectures have more difﬁ-culty in correctly understanding/grounding the spatial relations required to place blocks correctly, especially when the blocks are not easily identiﬁable.

Subfields: Artificial Intelligence, Control and Systems Engineering, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks (2019, background): “There are several existing benchmarks based on simple block worlds and fully observable scenes [9, 33].”
A Survey of Reinforcement Learning Informed by Natural Language (2019, methodology): “Human-generated natu-ral language instructions are used in [ MacMahon et al. , 2006; Bisk et al. , 2016; Misra et al. , 2017; Janner et al. , 2018; Chen et al. , 2018; Anderson et al. , 2018; Goyal et al. , 2019; Wang et al. , 2019 ] .”
Mapping Instructions and Visual Observations to Actions with Reinforcement Learning (2017, methodology): “We evaluate with the block world environment and data of Bisk et al. (2016), where each instruction moves one block (Figure 1).”
CLIPort: What and Where Pathways for Robotic Manipulation (2021, background): “Several works have proposed systems for instructing robots with natural language [7, 8, 9, 10, 41, 42, 43, 44, 45, 46, 47].”
Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation (2022, methodology): “Several works have proposed methods for grounding language in robot actions [49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60].”
Improving Natural Language Interaction with Robots Using Advice (2019, methodology): “We follow the deﬁnition by (Bisk et al., 2016) and due to space constraints refer the reader to that paper.”
Inferring Rewards from Language in Context (2022, background): “…2016; Fried et al., 2018a; Arumugam et al., 2017; Suhr et al., 2018) or map directly to primitive actions (Branavan et al., 2009; Andreas and Klein, 2015; Mei et al., 2016; Bisk et al., 2016; Misra et al., 2017; Guu et al., 2017; Suhr and Artzi, 2018; Anderson et al., 2018; Shridhar et al., 2020).”
Interactive Visual Grounding of Referring Expressions for Human-Robot Interaction (2018, methodology): “We make the usual assumption of binary relations [3, 18, 25].”
Opportunistic Active Learning for Grounding Natural Language Descriptions (2017, background): “Finally, by performing more complicated language understanding than simple stopword removal, future work may also be able to learn to jointly understand language predicates and human commands (as in [21, 22, 23]), additionally leveraging opportunistic active learning to ask questions that best clari”
A Corpus of Natural Language for Visual Reasoning (2017, background): “Visual reasoning has also been addressed in instructional language corpora (e.g., MacMahon et al., 2006; Chen and Mooney, 2011; Bisk et al., 2016), where executable instructions are grounded in manipulable environments.”
CHALET: Cornell House Agent Learning Environment (2018, unknown): “Simulators have also been used to evaluate natural language instruction following [14, 3, 13, 9, 8] and question answering [7, 6, 10, 19].”
Vision-Based Navigation With Language-Based Assistance via Imitation Learning With Indirect Intervention (2018, background): “Simple simulators as Puddle World Navigation [31] and Rich 3D Blocks World [5, 4] have facilitated understanding of fundamental representational and grounding issues by allowing for fast experimentation in easily-managed environments.”
Interactive Language: Talking to Robots in Real Time (2022, unknown): “Hindsight Language Labeling BLOCKS [50], [51] 30 n/r 7 7 3”
Interactive Learning from Activity Description (2021, background): “…2017; Goyal et al., 2019; Fu et al., 2019; Sumers et al., 2020), learning to ground language to actions (Chen & Mooney, 2011; Misra et al., 2014; Bisk et al., 2016; Liu et al., 2016; Wang et al., 2016; Li et al., 2017; 2020a;b), or devising EM-based algo-Interactive rithms to parse language…”
How to Motivate Your Dragon: Teaching Goal-Driven Agents to Speak and Act in Fantasy Worlds (2020, background): “Visual and embodied agents using natural language instructions (Bisk et al., 2016; Kolve et al., 2017; Anderson et al., 2018) or in language-based action spaces (Das et al.”
A Benchmark for Systematic Generalization in Grounded Language Understanding (2020, background): “, [31, 39, 22, 6], since the issues studied in gSCAN feature prominently in realistic NLU tasks such as teaching autonomous agents by demonstration.”
Efficient grounding of abstract spatial concepts for natural language interaction with robot platforms (2018, unknown): “Bisk et al. (2016) presented a neural network model that integrates plan inference within the language modeling framework, enabling task execution that is more robust to identiﬁcation errors and underspeciﬁcation of commands.”
Virtual Embodiment: A Scalable Long-Term Strategy for Artificial Intelligence Research (2016, background): “There has been work in linguistic grounding that allows for temporal aspects, for instance in videos [20, 21, 22], and both time and interaction, notably in the ﬁeld of robotics [23, 24, 25].”
Prospection: Interpretable plans from language by predicting the future (2019, background): “be used to generate task plans. Complementary to our work is the growing literature in NLP focusing on complex grounding instructions with esoteric references and other long tail linguistic phenomena [21], [22]. Natural language communication with robotics also allows for learning joint multimodal r”