Learning Hierarchical Teaching Policies for Cooperative Agents
By Dong-Ki Kim, Miao Liu, Shayegan Omidshafiei, Sebastian Lopez-Cot, Matthew Riemer, Golnaz Habibi, G. Tesauro, Sami Mourad, Murray Campbell, Jonathan P. How. 2019
33 citations recorded by the source corpus.
Collective learning can be greatly enhanced when agents effectively exchange knowledge with their peers. In particular, recent work studying agents that learn to teach other teammates has demonstrated that action advising accelerates team-wide learning. However, the prior work has simplified the learning of advising policies by using simple function approximations and only considered advising with primitive (low-level) actions, limiting the scalability of learning and teaching to complex domains. This paper introduces a novel learning-to-teach framework, called hierarchical multiagent teaching (HMAT), that improves scalability to complex environments by using the deep representation for student policies and by advising with more expressive extended action sequences over multiple levels of temporal abstraction. Our empirical evaluations demonstrate that HMAT improves team-wide learning progress in large, complex domains where previous approaches fail. HMAT also learns teaching policies that can effectively transfer knowledge to different teammates with knowledge of different tasks, even when the teammates have heterogeneous action spaces.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Matthew Riemer
- Irina Rish
- Miao Liu
- Khimya Khetarpal
- G. Tesauro
- Jonathan P. How
- Doina Precup
- Sergey Levine
Citations in the corpus
- Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “For example, agents can teach other agents as they learn so that each learns to better perform tasks that are required of them (Torrey and Taylor, 2013; Omidshafiei et al., 2019; Kim et al., 2020).”
- Influencing Long-Term Behavior in Multiagent Reinforcement Learning (2022, unknown): “Example frameworks include the studies regarding centralized training with decentralized execution, which account for other agents’ actions through centralized critics (Lowe et al., 2017; Foerster et al., 2018b; Yang et al., 2018; Omidshafiei et al., 2019; Iqbal & Sha, 2019; Kim et al., 2020).”
- Context-Specific Representation Abstraction for Deep Option Learning (2021, background): “…(Sutton, Precup, and Singh 1999; Bacon, Harb, and Precup 2017; Omidshafiei et al. 2018; Riemer, Liu, and Tesauro 2018; Riemer et al. 2020) and goal-conditioned hierarchies to learn abstract sub-goals (Nachum et al. 2019; Levy, Platt, and Saenko 2019; Kulkarni and Narasimhan 2016; Kim et al. 2020).”