Executing Instructions in Situated Collaborative Interactions

By Alane Suhr, Claudia Yan, Jack Schluger, Stanley Yu, H. Khader, Marwa Mouallem, Iris Zhang, Yoav Artzi. 2019

99 citations recorded by the source corpus.

We study a collaborative scenario where a user not only instructs a system to complete tasks, but also acts alongside it. This allows the user to adapt to the system abilities by changing their language or deciding to simply accomplish some tasks themselves, and requires the system to effectively recover from errors as the user strategically assigns it new goals. We build a game environment to study this scenario, and learn to map user instructions to system actions. We introduce a learning approach focused on recovery from cascading errors between instructions, and modeling methods to explicitly reason about instructions with multiple goals. We evaluate with a new evaluation protocol using recorded interactions and online games with human users, and observe how users adapt to the system abilities.

Subfields: Artificial Intelligence, Natural Language Processing, Dialogue & Conversational AI

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Dialogue & Conversational AI

Citations in the corpus

LanCon-Learn: Learning with Language to Enable Generalization in Multi-Task Manipulation (2021, background): “The instruction-following literature includes several examples of language-based learning agents, centered on navigation [31], [32], [33], [10], [9], [34], [12], [35] or simple manipulation tasks [11], [36], [22].”
Towards Ecologically Valid Research on Language User Interfaces (2020, unknown): “Another example is CerealBar (Suhr et al., 2019), where two agents, a leader and a follower, navigate a toy 3D environment in order to collect a sets of cards.”
Learning to Map Natural Language Instructions to Physical Quadcopter Control using Simulated Flight (2019, methodology): “This output is used to compute two distributions, and can be increased if more distribution are predicted, such as in Suhr et al. [44].”
Learning Rewards from Linguistic Feedback (2020, background): “…other domains including action coordination (Djalali et al. 2011; Djalali, Lauer, and Potts 2012; Potts 2012; Ilinykh, Zarrieß, and Schlangen 2019; Suhr et al. 2019), reference pragmatics (He et al. 2017; Udagawa and Aizawa 2019), navigation (Thomason et al. 2019), and “Wizard of Oz” experiments…”
Audio-Visual Embodied Navigation (2019, background): “Visual navigation can be tied with other tasks to attain intelligent behavior, such as question answering [23, 25, 39, 99], active visual recognition [50, 110, 109], instruction following [88, 6, 86, 17, 16] and collaborative tasks with visual inputs [49, 48, 24].”
MindCraft: Theory of Mind Modeling for Situated Dialogue in Collaborative Tasks (2021, background): “…MindCraft including in custom 2D worlds (Liu et al., 2012; Udagawa and Aizawa, 2020, 2019), in the physical world with human-robot interactions (McGuire et al., 2002; Chai et al., 2014, 2018; Thomason et al., 2020), and in various 3D virtual worlds (Bisk et al., 2018; Suhr et al., 2019).”
A Cordial Sync: Going Beyond Marginal Policies for Multi-Agent Embodied Tasks (2020, background): “Embodied navigation also aids AI agents to develop behavior such as instruction following [38,4,82,95,3], city navigation [18,64,63,94], question answering [21,22,34,97,24], and active visual recognition [105,104].”