Accelerating Reinforcement Learning through Implicit Imitation

By Craig Boutilier, Bob Price. 2003

204 citations recorded by the source corpus.

Imitation can be viewed as a means of enhancing learning in multiagent environments. It augments an agent's ability to learn useful behaviors by making intelligent use of the knowledge implicit in behaviors demonstrated by cooperative teachers or other more experienced agents. We propose and study a formal model of implicit imitation that can accelerate reinforcement learning dramatically in certain cases. Roughly, by observing a mentor, a reinforcement-learning agent can extract information about its own capabilities in, and the relative value of, unvisited parts of the state space. We study two specific instantiations of this model, one in which the learning agent and the mentor have identical abilities, and one designed to deal with agents and mentors with difierent action sets. We illustrate the benefits of implicit imitation by integrating it with prioritized sweeping, and demonstrating improved performance and convergence through observation of single and multiple mentors. Though we make some stringent assumptions regarding observability and possible interactions, we briefly comment on extensions of the model that relax these restricitions.

Subfields: Artificial Intelligence, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

Citations in the corpus

Multi-agent reinforcement learning as a rehearsal for decentralized planning (2016, neutral): “For instance, it differs from imitation learning [23] in that the external observer is not a mentor and does not participate in the decision process.”
Teaching on a budget: agents advising agents in reinforcement learning (2013, background): “Other examples include imitation learning [8], in which a student learns by observing a teacher, apprentice learning [3], in which a student asks a teacher for advice whenever its confidence in a state is low, and advice exchange [7], in which peers ask for advice from each other based on heuristics”
Reinforcement learning from simultaneous human and MDP reward (2012, methodology): “Learning from demonstration (LfD) has also been used to improve reinforcement learning, using preprogrammed policies [11] or humans [17, 12] to provide demonstrations for an agent that observes and learns.”
Behavior transfer for value-function-based reinforcement learning (2005, background): “While it is well understood how to add this type of guidance to a learner [13], we would prefer to allow the agent to learn faster by training on different (perhaps pre-existing) tasks rather than creating artiﬁcial tasks that are easier to learn.”
Multiagent Collaborative Task Learning through Imitation (2007, background): “Price and Boutilier [21] present a multiagent system in which novice agents learn by passively observing other agents in the environment .”
Abstraction Levels for Robotic Imitation: Overview and Computational Approaches (2010, background): “For instance, it can speed up learning either by providing an initial solution for the intended task that can then be improved by trial-and-error [109] or by guiding exploration [112, 114].”
Transfer in variable-reward hierarchical reinforcement learning (2008, background): “Researchers have also looked at the problem of transferring knowledge from one learner to another as imitation ( Price and Boutilier 2003 ).”
On the use of hybrid reinforcement learning for autonomic resource allocation (2007, result): “[21–23]), however, we note that the use of batch training is one aspect in which our approach differs from previous studies of such combinations.”
Exploration from Demonstration for Interactive Reinforcement Learning (2016, background): “There exists is a wide variety of work in the ﬁeld of Inter-active Machine Learning, namely Learning by Demonstration [4], Imitation Learning [30], Policy Shaping [17] and TAMER [22].”
Combining manual feedback with subsequent MDP reward signals for reinforcement learning (2010, unknown): “Imitation learning, or programming by design, has also been used to improve learning within reinforcement learning [13].”
HG-DAgger: Interactive Imitation Learning with Human Experts (2018, methodology): “Data is gathered from an expert policy and is used to train a novice policy [1], [2].”
A Comprehensive Survey of Multiagent Reinforcement Learning (2008, unknown): “exploit the potential of these benefits [41], [53].”
A Multi-Facet Survey on Memetic Computation (2011, background): “In implicit imitation [192] or teacher-based supervised evolutionary learning [57], beginner agents learn by imitating the memes from expert agents of the same environment, based on the payoff criterion.”
Probabilistic policy reuse in a reinforcement learning agent (2006, background): “Different knowledge sources can be used, for example through learning policies by imitation [10].”
Multi-agent Reinforcement Learning: An Overview (2010, background): “For instance, the agents can exchange information using communication [123], skilled agents may serve as teachers for the learner [30], or the learner may watch and imitate the skilled agents [95].”
A Bayesian Approach for Policy Learning from Trajectory Preference Queries (2012, background): “via imitation learning [15, 3, 13] or inverse reinforcement learning [12].”