Observational Learning by Reinforcement Learning

By Diana Borsa, Bilal Piot, Siqi Liu, R´emi Munos, Olivier Pietquin. 2017

51 citations recorded by the source corpus.

Observational learning is a type of learning that occurs as a function of observing, retaining and possibly imitating the behaviour of another agent. It is a core mechanism appearing in various instances of social learning and has been found to be employed in several intelligent species, including humans. In this paper, we investigate to what extent the explicit modelling of other agents is necessary to achieve observational learning through machine learning. Especially, we argue that observational learning can emerge from pure Reinforcement Learning (RL), potentially coupled with memory. Through simple scenarios, we demonstrate that an RL agent can leverage the information provided by the observations of an other agent performing a task in a shared environment. The other agent is only observed through the effect of its actions on the environment and never explicitly modeled. Two key aspects are borrowed from observational learning: i) the observer behaviour needs to change as a result of viewing a 'teacher' (another agent) and ii) the observer needs to be motivated somehow to engage in making use of the other agent's behaviour. The later is naturally modeled by RL, by correlating the learning agent's reward with the teacher agent's behaviour.

Subfields: Artificial Intelligence, Sociology and Political Science, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Sociology and Political Science

In Reinforcement Learning

Citations in the corpus

Imitating Interactive Intelligence (2020, background): “Further, it is at present intrinsically difficult to do so because algorithms and systems have not yet been developed that can perceive and understand the intentions of humans in a way that transfers across radical changes in embodiment, environment, and perspective (Stadie et al., 2017; Borsa et al”
Hybrid Reinforcement Learning with Expert State Sequences (2019, methodology): “On the other hand, IRL methods aim to infer the goal of an expert, expressed as a reward function (Ng and Russell 2000; Abbeel and Ng 2004; Ziebart et al. 2008; Levine, Popovic, and Koltun 2011; Borsa et al. 2017).”
Reward is enough (2021, background): “…driven by the maximisation of reward, from the perspective of a single agent that simply observes other agents as integral parts of its environment [6], potentially leading to many of the same beneﬁts as behavioural cloning – such as the sample-eﬃcient acquisition of knowledge – but in a much…”
Learning to Interactively Learn and Assist (2019, background): “…training; either continuously (Ross, Gordon, and Bagnell 2011b; Kelly et al. 2018) or at the agent’s discretion (Ross, Gordon, and Bagnell 2011a; Borsa et al. 2017; Xu et al. 2018; Hester et al. 2018; James, Bloesch, and Davi-son 2018; Yu et al. 2018a; Krening 2018; Brown, Cui, and Niekum…”