Language to Action: Towards Interactive Task Learning with Physical Agents

By Joyce Yue Chai, Qiaozi Gao, Lanbo She, Shaohua Yang, S. Saba-Sadiya, Guangyue Xu. 2018

98 citations recorded by the source corpus.

Language communication plays an important role in human learning and knowledge acquisition. With the emergence of a new generation of cognitive robots, empowering these robots to learn directly from human partners becomes increasingly important. This paper gives a brief introduction to interactive task learning where humans can teach physical agents new tasks through natural language communication and action demonstration. It discusses research challenges and opportunities in language and communication grounding that are critical in this process. It further highlights the importance of commonsense knowledge, particularly the very basic physical causality knowledge, in grounding language to perception and action.

Subfields: Artificial Intelligence, LLM Agents & Tool Use, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Mohit Bansal
2. Percy Liang
3. Yejin Choi
4. Luke S. Zettlemoyer
5. Caiming Xiong
6. Richard Socher
7. Graham Neubig
8. Jianfeng Gao

In LLM Agents & Tool Use

1. Jacob Andreas
2. Hal Daumé Iii
3. Yonatan Bisk
4. Hadas Kress-Gazit
5. Mirella Lapata
6. Yoshua Bengio
7. Caiming Xiong
8. Nakul Gopalan

In Multimodal Learning

1. Caiming Xiong
2. Yoshua Bengio
3. Angeliki Lazaridou
4. Mirella Lapata
5. Richard Socher
6. Hal Daumé Iii
7. Yonatan Bisk
8. Ari Holtzman

Citations in the corpus

• Experience Grounds Language (2020, background): “, 2020) or map language to physical robot control (Blukis et al., 2019; Chai et al., 2018).”
• Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completion (2021, background): “That focus yields language descriptions of object targets for manipulation such as put the strawberries on the cutting board and slice them into pieces (Chai et al., 2018).”
• Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learning (2019, background): “On the other hand, experimentation on real humans and robots, despite expensive and time-consuming, are important for understanding the true complexity of real-world scenarios (Chai et al., 2018, 2016; Ryb-ski et al., 2007; Mohan and Laird, 2014; Liu et al., 2016; She et al., 2014).”
• MindCraft: Theory of Mind Modeling for Situated Dialogue in Collaborative Tasks (2021, background): “…MindCraft including in custom 2D worlds (Liu et al., 2012; Udagawa and Aizawa, 2020, 2019), in the physical world with human-robot interactions (McGuire et al., 2002; Chai et al., 2014, 2018; Thomason et al., 2020), and in various 3D virtual worlds (Bisk et al., 2018; Suhr et al., 2019).”
• ESPRIT: Explaining Solutions to Physical Reasoning Tasks (2020, background): “Several recent papers propose to use natural language for explanation and commonsense reasoning (Lei et al., 2016; Camburu et al., 2018; Forbes and Choi, 2017; Chai et al., 2018; Forbes et al., 2019; Rajani et al., 2019; DeYoung et al., 2020).”
• Robots That Use Language (2020, background): “A more direct approach is to learn the spatial semantics of actions directly from language (187) (shown in Figure 1b).”