Teaching on a budget: agents advising agents in reinforcement learning

By Lisa A. Torrey, Matthew E. Taylor. 2013

202 citations recorded by the source corpus.

This paper introduces a teacher-student framework for reinforcement learning. In this framework, a teacher agent instructs a student agent by suggesting actions the student should take as it learns. However, the teacher may only give such advice a limited number of times. We present several novel algorithms that teachers can use to budget their advice effectively, and we evaluate them in two experimental domains: Mountain Car and Pac-Man. Our results show that the same amount of advice, given at different moments, can have different effects on student learning, and that teachers can significantly affect student learning even when students use different learning methods and state representations.

Subfields: Artificial Intelligence, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

Citations in the corpus

Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “For example, agents can teach other agents as they learn so that each learns to better perform tasks that are required of them (Torrey and Taylor, 2013; Omidshafiei, Kim, Liu, Tesauro, Riemer, Amato, Campbell, and How, 2019; Kim, Liu, Omidshafiei, Lopez-Cot, Riemer, Habibi, Tesauro, Mourad,…”
Residual Reinforcement Learning for Robot Control (2018, background): “Other work has focused on human feedback [17], [18], [19], [20], [21] to inform the agent about rewards or to encourage safety.”
A Q-values Sharing Framework for Multi-agent Reinforcement Learning under Budget Constraint (2020, background): “In order to overcome the challenge of having no fixed roles of teacher and student, they extend the heuristics from [4] to measure an agent’s confidence in a given state based on the number of times that it visits the state.”
A Survey of Learning in Multiagent Environments: Dealing with Non-Stationarity (2017, background): “Giving advice to agents (Torrey and Taylor, 2013; Zhan et al., 2016) is another example in which multiagent algorithms are still in an early stage of development (da Silva et al.”
Learning Hierarchical Teaching Policies for Cooperative Agents (2019, background): “One teacher action in this extended view corresponds to providing multiple pieces of advice during one episode, which contrasts with previous teaching approaches that the task-level policies were updated based on a single piece of advice [1, 4, 6, 27, 36].”
Learning to Teach in Cooperative Multiagent Reinforcement Learning (2018, background): “Importance Advising and Correct Important heuristics lead agents to suboptimal (distant) goals in Hallway and Room, yet attain positive value due to domain symmetries.”
Online Transfer Learning in Reinforcement Learning Domains (2015, methodology): “This paper focuses on action advice Torrey and Taylor (2013): as the student agent practices, the teacher agent suggests actions to take.”
Agent-Agnostic Human-in-the-Loop Reinforcement Learning (2017, background): “Prior literature has investigated how people can help RL agents learn more efﬁciently through different methods of interaction [32, 24, 28, 36, 42, 48, 21, 17, 25, 43, 23, 49, 46, 44, 30, 10].”
Parallel Knowledge Transfer in Multi-Agent Reinforcement Learning (2020, background): “In teacher-initiated approaches, teachers decide when to teach based on the comparison between student’s and teacher’s learning experience, such as Importance Advising [16], Early Correcting [15], and Correct Important [16].”
A Q-values Sharing Framework for Multiple Independent Q-learners (2019, unknown): “One notable approach is the teacher-student framework [8].”
A Survey of Preference-Based Reinforcement Learning Methods (2017, methodology): “Advice-based algorithms (Torrey and Taylor, 2013; Knox and Stone, 2010, 2012; Griffith et al., 2013) differ from PbRL in that the (possibly preference-based) advice is given in addition to a numeric reward signal.”