Environment-Driven Lexicon Induction for High-Level Instructions

By Dipendra Kumar Misra, Kejia Tao, Percy Liang, Ashutosh Saxena. 2015

57 citations recorded by the source corpus.

We focus on the task of interpreting complex natural language instructions to a robot, in which we must ground high-level commands such as microwave the cup to low-level actions such as grasping. Previous approaches that learn a lexicon during training have inadequate coverage at test time, and pure search strategies cannot handle the exponential search space. We propose a new hybrid approach that leverages the environment to induce new lexical entries at test time, even for new verbs. Our semantic parsing model jointly reasons about the text, logical forms, and environment over multi-stage instruction sequences. We introduce a new dataset and show that our approach is able to successfully ground new verbs such as distribute, mix, arrange to complex logical forms, each containing up to four predicates.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents (2022, background): “A series of them parse language instructions into formal logic or rely mainly on lexical analysis to resolve various linguistic ambiguities for embodied agents [2, 28, 29, 44].”
Mapping Instructions and Visual Observations to Actions with Reinforcement Learning (2017, methodology): “Misra (Misra et al., 2015) provides a parameterized action space (e.g., grasp(cup) ), which leads to a large number of potential actions.”
Finding "It": Weakly-Supervised Reference-Aware Visual Grounding in Instructional Videos (2018, background): “During training, we leverage the Stanford CoreNLP parser [30] to automatically parse entities.”
Language to Action: Towards Interactive Task Learning with Physical Agents (2018, methodology): “Using the data made available by [Misra et al. , 2015], our empirical results have shown the hypothesis space representation of grounded semantics signiﬁcantly outperforms the single hypothesis representation.”
Grounding Predicates through Actions (2021, background): “Other interesting avenues of work include using natural language to learn the pre- and post-conditions of actions [38, 39], or combining visual groundings of symbols with natural language [3, 40].”
A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Execution (2021, background): “In contrast, we focus on high-level instructions, which are more likely in home environments [39].”
Learning to Learn Semantic Parsers from Natural Language Supervision (2019, background): “…of systems asking for and conﬁrming information (Artzi and Zettle-moyer, 2011; Thomason et al., 2015; Padmakumar et al., 2017); and learning from interactions with a simulated world environment (Branavan et al., 2009; Artzi and Zettlemoyer, 2013; Goldwasser and Roth, 2014; Misra et al., 2015).”
Collaborative Language Grounding Toward Situated Human-Robot Dialogue (2017, methodology): “Using data made available by Misra et al. (2015), our empirical results have shown that the hypothesis space representation significantly outperforms the representation with single hypothesis of goal state.”
Mapping natural language commands to web elements (2018, methodology): “Our task is inspired by the semantic parsing literature, which aims to map natural language utterances into actions such as database queries and object manipulation (Zelle and Mooney, 1996; Chen and Mooney, 2011; Artzi and Zettlemoyer, 2013; Berant et al., 2013; Misra et al., 2015; Andreas and Klein”
Mapping Instructions to Actions in 3D Environments with Visual Goal Prediction (2018, background): “Mapping instruction to action has been studied extensively with intermediate symbolic representations (e.g., Chen and Mooney, 2011; Kim and Mooney, 2012; Artzi and Zettlemoyer, 2013; Artzi et al., 2014; Misra et al., 2015, 2016).”
Neural Enquirer: Learning to Query Tables (2015, background): “The parsers, designed towards this new learning paradigm, take diﬀerent types of forms, ranging from generic chart parsers [3, 11] to more speciﬁcally engineered, task-oriented ones [12, 8].”
CHALET: Cornell House Agent Learning Environment (2018, unknown): “Simulators have also been used to evaluate natural language instruction following [14, 3, 13, 9, 8] and question answering [7, 6, 10, 19].”
Reward Learning from Narrated Demonstrations (2018, background): “For instance, paired examples of instruction and action sequences have been collected through Amazon Mechanical Turk [8, 36, 37, 52].”