Understanding the origin of information-seeking exploration in probabilistic objectives for control

By Beren Millidge, Alexander Tschantz, Anil K. Seth, C. L. Buckley. 2021

13 citations recorded by the source corpus.

The exploration-exploitation trade-off is central to the description of adaptive behaviour in fields ranging from machine learning, to biology, to economics. One popular approach to solving this trade-off has been to propose that agents possess an intrinsic ’exploratory drive’ whereby agent’s are driven to maximize the information they gain about the world through action, as well as the more traditional utility maximization objective. In this paper we mathematically investigate the nature of such objectives and demonstrate that the combination of utility maximizing and information-seeking behaviour arises from the minimization of a class of objectives which we call divergence objectives. We propose a dichotomy in the objective functions underlying adaptive behaviour between evidence objectives, which correspond to well-known reward or utility maximizing objectives and our novel divergence objectives which instead seek to minimize the divergence between the agent’s expected and desired distribution over futures and which give rise to information-seeking exploration. This new class of divergence objectives provides the mathematical foundation for a much richer understanding of the exploratory components of adaptive and intelligent action, beyond simply greedy utility maximization, and can also provide a mathematically elegant explanation for anomalous behaviour often observed in psychophysical and choice tasks such as probability matching.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Reinforcement Learning