Interactive Learning from Activity Description
By Khanh Nguyen, Dipendra Kumar Misra, Robert E. Schapire, Miroslav Dud'ik, Patrick Shafto. 2021
36 citations recorded by the source corpus.
We present a novel interactive learning protocol that enables training request-fulfilling agents by verbally describing their activities. Unlike imitation learning (IL), our protocol allows the teaching agent to provide feedback in a language that is most appropriate for them. Compared with reward in reinforcement learning (RL), the description feedback is richer and allows for improved sample complexity. We develop a probabilistic framework and an algorithm that practically implements our protocol. Empirical results in two challenging request-fulfilling problems demonstrate the strengths of our approach: compared with RL baselines, it is more sample-efficient; compared with IL baselines, it achieves competitive success rates without requiring the teaching agent to be able to demonstrate the desired behavior using the learning agent's actions. Apart from empirical evaluation, we also provide theoretical guarantees for our algorithm under certain assumptions about the teacher and the environment.
Subfields: Reinforcement Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Reinforcement Learning
- Shunyu Yao
- Kyunghyun Cho
- Cédric Colas
- Karthik Narasimhan
- Clément Moulin-Frier
- Weizhu Chen
- Tomasz Korbak
- Theodore R. Sumers
Citations in the corpus
- Training Language Models with Language Feedback (2022, background): “, 2019), or purely with language by providing descriptions of trajectories (Nguyen et al., 2021).”
- Cognitive Architectures for Language Agents (2023, background): “Classic linguistic interactions allow the agent to accept instructions (Winograd, 1972; Tellex et al., 2011; Chen and Mooney, 2011; Bisk et al., 2016) or learn from people (Nguyen et al., 2021; Sumers et al., 2022; 2021; Wang et al., 2016).”
- Training Language Models with Language Feedback at Scale (2023, background): “…inferring reward functions (Lin et al., 2022; Sumers et al., 2021; Fidler et al., 2017, inter alia ), and training a model via strong supervision (Andreas et al., 2017; Kaplan et al., 2017), reward shaping (Goyal et al., 2019), or by providing descriptions of trajectories (Nguyen et al., 2021).”
- Vygotskian Autotelic Artificial Intelligence: Language and Culture Internalization for Human-Like AI (2022, background): “120 In line with Vygotsky’s theory, these agents internalize a model of a descriptive social partner.”
- CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing (2023, background): “[23] Khanh X Nguyen, Dipendra Misra, Robert Schapire, Miroslav Dudík, and Patrick Shafto.”