Jointly Improving Parsing and Perception for Natural Language Commands through Human-Robot Dialog

By Jesse Thomason, Aishwarya Padmakumar, Jivko Sinapov, Yuqian Jiang, Harel Yedidsion, Justin W. Hart, Peter Stone, Raymond J. Mooney. 2020

60 citations recorded by the source corpus.

In this work, we present methods for using human-robot dialog to improve language understanding for a mobile robot agent. The agent parses natural language to underlying semantic meanings and uses robotic sensors to create multi-modal models of perceptual concepts like red and heavy. The agent can be used for showing navigation routes, delivering objects to people, and relocating objects from one location to another. We use dialog clari_cation questions both to understand commands and to generate additional parsing training data. The agent employs opportunistic active learning to select questions about how words relate to objects, improving its understanding of perceptual concepts. We evaluated this agent on Amazon Mechanical Turk. After training on data induced from conversations, the agent reduced the number of dialog questions it asked while receiving higher usability ratings. Additionally, we demonstrated the agent on a robotic platform, where it learned new perceptual concepts on the y while completing a real-world task.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Experience Grounds Language (2020, background): “world (Tellex et al., 2011; Matuszek, 2018; Thomason et al., 2020; Tellex et al., 2020) must translate from language to action.”
Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (2022, methodology): “This enables a more flexible framework and more general abstract, tasks than semantic parsing [43, 46, 56].”
Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completion (2021, background): “…misses the full range of experience that can ground language in the world (Bisk et al., 2020), such as haptic feedback during object manipulation (Thomason et al., 2020, 2016; Sinapov et al., 2014), and audio (Chen et al., 2020a) and speech (Harwath et al., 2019; Ku et al., 2020) features of the…”
Code as Policies: Language Model Programs for Embodied Control (2022, unknown): “, semantic parsing [25], [27]–[32]), planning [14], [17], [18], and low-level policies (e.”
MindCraft: Theory of Mind Modeling for Situated Dialogue in Collaborative Tasks (2021, background): “…MindCraft including in custom 2D worlds (Liu et al., 2012; Udagawa and Aizawa, 2020, 2019), in the physical world with human-robot interactions (McGuire et al., 2002; Chai et al., 2014, 2018; Thomason et al., 2020), and in various 3D virtual worlds (Bisk et al., 2018; Suhr et al., 2019).”
Learning Rewards from Linguistic Feedback (2020, background): “Dialogue systems (Artzi and Zettlemoyer 2011; Li et al. 2016) learn to interpret user queries in the context of the ongoing interaction, while robots and assistants (Thomason et al. 2015; Wang et al. 2019; Thomason et al. 2020; Szlam et al. 2019) ground language in their physical surroundings.”