Walk the Talk: Connecting Language, Knowledge, and Action in Route Instructions

By M. MacMahon, Brian J. Stankiewicz, Benjamin Kuipers. 2006

431 citations recorded by the source corpus.

Subfields: Natural Language Processing, LLM Agents & Tool Use

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In LLM Agents & Tool Use

Citations in the corpus

Experience Grounds Language (2020, background): “An embodied agent, whether in a virtual world, such as a 2D Maze (MacMahon et al., 2006), a Baby AI in a grid world (Chevalier-Boisvert et al., 2019), Vision-and-Language Navigation (Anderson et al., 2018; Thomason et al., 2019b), or the real world (Tellex et al., 2011; Matuszek, 2018; Thomason et a”
Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (2022, background): “There is a long history of research studying how to connect language and behavior [46, 47, 48, 49, 50].”
ALFWorld: Aligning Text and Embodied Environments for Interactive Learning (2020, background): “1 Unlike prior work on instruction following (MacMahon et al., 2006; Anderson et al., 2018a), which typically uses a ﬁxed corpus of cross-modal expert demonstrations, we argue that aligned parallel environments like ALFWorld offer a distinct advantage: they allow agents to explore, interact , and…”
ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks (2019, background): “In vision-and-language navigation tasks, either natural or templated language describes a route to a goal location through egocentric visual observations [3, 12, 13, 14, 30].”
Vision-and-Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments (2017, background): “cally achieved either by assuming that the set of all navigation goals, or objects will be identiﬁed by label [13,52], or by operating in visually restricted environments requiring limited perception [12,20,24,29,35,38,55]. Our work contributes for the ﬁrst time a navigation benchmark dataset that i”
A Corpus for Reasoning about Natural Language Grounded in Photographs (2018, background): “…and Parikh, 2013; Antol et al., 2015), caption generation (Chen et al., 2016), referring expression resolution (Mitchell et al., 2010; Matuszek et al., 2012; FitzGerald et al., 2013), and instruction following (MacMahon et al., 2006; Chen and Mooney, 2011; Bisk et al., 2016; Misra et al., 2018).”
Weakly Supervised Learning of Semantic Parsers for Mapping Instructions to Actions (2013, methodology): “For example, in the navigation domain (MacMahon et al., 2006), S is a set of positions on a map.”
Understanding Natural Language Commands for Robotic Navigation and Mobile Manipulation (2011, background): “Beginning with SHRDLU (Winograd, 1970), many systems have exploited the compositional structure of language to statically generate a plan corresponding to a natural language command (Dzifcak et al., 2009; Hsiao et al., 2008; MacMahon, Stankiewicz, and Kuipers, 2006; Skubic et al., 2004).”
Learning to Interpret Natural Language Navigation Instructions from Observations (2011, methodology): “To execute a navigation plan, we ﬁrst transform it from our formal representation into a compound action speciﬁca-tion and then use M ARCO ’s executor to carry out the actions in the virtual worlds.”
A Roadmap Towards Machine Intelligence (2015, background): “We have several points of contact with the semantic parsing literature, such as navigation tasks in an artificial world (MacMahon et al., 2006) and reward-based learning from natural language instructions (Chen and Mooney, 2011; Artzi and Zettlemoyer, 2013).”
Mapping Instructions and Visual Observations to Actions with Reinforcement Learning (2017, methodology): “8 The most similar dataset is SAIL, which provides only partial observability of the environment (i.e., the agent observes what is around it only).”
BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning (2018, background): “There has been much research on instruction following with natural language (Tellex et al., 2011; Chen and Mooney, 2011; Artzi and Zettlemoyer, 2013; Mei et al., 2016; Williams et al., 2018) as well as several datasets including SAIL (Macmahon et al., 2006; Chen and Mooney, 2011) and Room-to-Room (A”
Vision-and-Dialog Navigation (2019, methodology): “Dataset Language Context Visual Context Human Amb UnderS Temporal Real-world Temporal MARCO[8, 9], DRIF[10] 3 7 7 Single Instruction 7 Dynamic R2R[7], Touchdown[11] 3 7 7 Single Instruction 3 Dynamic EQA[12], IQA[13] 7 7 3 Single Question 7 Dynamic CLEVR[14] 7 7 - Single Question 7 Static VQA[15, 16”