Learning to Parse Natural Language Commands to a Robot Control System

By Cynthia Matuszek, Evan V. Herbst, Luke S. Zettlemoyer, Dieter Fox. 2012

423 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Language Understanding for Text-based Games using Deep Reinforcement Learning (2015, methodology): “, 2010), navigating with directions (Vogel and Jurafsky, 2010; Kollar et al., 2010; Artzi and Zettlemoyer, 2013; Matuszek et al., 2013) and playing computer games (Eisenstein et al.”
Parsing Algebraic Word Problems into Equations (2015, background): “More closely related is work on language grounding, whose goal is the interpretation of a sentence in the context of a world representation (Branavan et al., 2009; Liang et al., 2009; Chen et al., 2010; Bordes et al., 2010; Feng and Lapata, 2010; Hajishirzi et al., 2011; Matuszek et al., 2012; Hajis”
LILA: Language-Informed Latent Actions (2021, background): “A variety of methods have sought to combine language and robotics, spanning approaches that map language to planning primitives [9, 30, 31, 32, 33], perform imitation learning from demonstrations and instructions [11, 34, 35, 36], and pair language instructions with reward functions for…”
A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Input (2014, background): “Integrated systems that execute commands : Others [9, 10, 11, 12, 13] focus on the task of learning the representation of natural language in the restricted setting of executing commands.”
Weakly Supervised Learning of Semantic Parsers for Mapping Instructions to Actions (2013, background): “There has also been work on learning for semantic analysis tasks from grounded data, including event streams (Liang et al., 2009; Chen et al., 2010) and language paired with visual perception (Matuszek et al., 2012).”
A Joint Model of Language and Perception for Grounded Attribute Learning (2012, background): “Most closely related to our work are approaches that learn probabilistic language models from natural language input (Matuszek et al., 2012; Chen & Mooney, 2011), especially those that include a visual component (Tellex et al., 2011).”
Mapping Instructions and Visual Observations to Actions with Reinforcement Learning (2017, methodology): “Existing approaches addressing this problem assume structured environment representations (e.g.,. Chen and Mooney, 2011; Mei et al., 2016), or combine separately trained models (e.g., Matuszek et al., 2010; Tellex et al., 2011), including for language understanding and visual reasoning.”
Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Text (2020, neutral): “…recent pipeline-based approaches use learning algorithms to map language to a program that can then interface with a planner [Chen and Mooney, 2011, Matuszek et al., 2013, Wang et al., 2016], and/or a controller, both of which may have priviledged information about how the world connects to the…”
Neural Modular Control for Embodied Question Answering (2018, background): “Recent deep reinforcement learning-based approaches to this explore it in 2D gridworlds [16, 20, 21], simple visual [22–27] and textual [28, 29] environments, perceptuallyrealistic 3D home simulators [1, 30–33], as well as real indoor scenes [34–36].”
Self-Monitoring Navigation Agent via Auxiliary Progress Estimation (2019, background): “…address the class of tasks where the agent does not have an explicit representation of the target (e.g., location in a map or image representation of the goal) to know if the goal has been reached or not (Matuszek et al., 2013; Hemachandra et al., 2015; Duvallet et al., 2016; Arkin et al., 2017).”
Cooking with Semantics (2014, methodology): “We assume access to a simple cooking simulator that can take in a stream of low-level instructions to produce a new state; this implements the world dynamics model p ( S t | S t − 1 , A t ) .”
A framework for learning semantic maps from grounded natural language descriptions (2014, methodology): “One class of solutions [41, 28, 8, 5, 31, 30] considers the problem as one of parsing free-form commands into their formal language equivalent, which a planner takes as input.”
Mapping Instructions to Actions in 3D Environments with Visual Goal Prediction (2018, background): “Goal prediction is related to referring expression interpretation (Matuszek et al., 2012a; Krish-namurthy and Kollar, 2013; Kazemzadeh et al., 2014; Kong et al., 2014; Yu et al., 2016; Mao et al., 2016; Kitaev and Klein, 2017).”
Interpreting multimodal referring expressions in real time (2016, background): “Batched interpretation is highly applicable in written communication [15, 6, 14, 21], but in recent years continuous interpretation has proved more valuable in the real-time domain.”
Correcting Robot Plans with Natural Language Feedback (2022, background): “However, language also brings with it ambiguity and requires a model of symbol grounding and spatio-temporal reasoning to relate the concepts expressed in language to the robot’s state and action spaces [17, 25, 37, 22, 3, 30, 29].”
Language to Rewards for Robotic Skill Synthesis (2023, background): “itives [29], while more recent work utilize end-to-end models that produce actions conditioned on natural language descriptions.”
Semantic Parsing for Text to 3D Scene Generation (2014, background): “Furthermore, work on parsing instructions to robots (Matuszek et al., 2013; Tellex et al., 2014) has analogues in the context of discourse about physical scenes.”
Real-time natural language corrections for assistive robotic manipulators (2017, background): “Similarly, Matuszek et al. (2013) learn a natural language-to-action model based on example pairs of commands and their corresponding control expressions.”