Learning Purposeful Behaviour in the Absence of Rewards

By Marlos C. Machado, Michael H. Bowling. 2016

32 citations recorded by the source corpus.

Artificial intelligence is commonly defined as the ability to achieve goals in the world. In the reinforcement learning framework, goals are encoded as reward functions that guide agent behaviour, and the sum of observed rewards provide a notion of progress. However, some domains have no such reward signal, or have a reward signal so sparse as to appear absent. Without reward feedback, agent behaviour is typically random, often dithering aimlessly and lacking intentionality. In this paper we present an algorithm capable of learning purposeful behaviour in the absence of rewards. The algorithm proceeds by constructing temporally extended actions (options), through the identification of purposes that are "just out of reach" of the agent's current behaviour. These purposes establish intrinsic goals for the agent to learn, ultimately resulting in a suite of behaviours that encourage the agent to visit different parts of the state space. Moreover, the approach is particularly suited for settings where rewards are very sparse, and such behaviours can help in the exploration of the environment until reward is observed.

Subfields: Artificial Intelligence, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

Citations in the corpus

Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation (2016, background): “In more recent work, Machado et al. [24] presented an option discovery algorithm where the agent is encouraged to explore regions that were previously out of reach.”
Deep Successor Reinforcement Learning (2016, background): “This enables learning the SR and reward function from raw sensory observations with end-to-end training.”
Unsupervised Control Through Non-Parametric Discriminative Rewards (2018, methodology): “Machado & Bowling (2016) proposes a method based on an eigendecomposition of differences in features between successive states, further explored and extended in Machado et al. (2017).”
Neural Program Lattices (2016, background): “More recent work has developed option discovery algorithms where the agent is encouraged to explore regions that were previously out of reach (Machado & Bowling, 2016) while other work has shown the benefits of manually chosen abstractions in large state spaces (Kulkarni et al., 2016).”
Discovering Options for Exploration by Minimizing Cover Time (2019, unknown): “…& Barto, 2001; Menache et al., 2002; Stolle & Precup, 2002; S¸ims¸ek & Barto, 2004; Simsek et al., 2005; S¸ims¸ek & Barto, 2009; Konidaris & Barto, 2009; Machado & Bowling, 2016; Kompella et al., 2017; Machado et al., 2017a;b; Eysenbach et al., 2018; Nair et al., 2018; Riedmiller et al., 2018).”