Unsupervised Real-Time Control Through Variational Empowerment
By Maximilian Karl, Maximilian Soelch, Philip Becker-Ehmck, Djalel Benbouzid, Patrick van der Smagt, Justin Bayer. 2017
60 citations recorded by the source corpus.
We introduce a methodology for efficiently computing a lower bound to empowerment, allowing it to be used as an unsupervised cost function for policy learning in real-time control. Empowerment, being the channel capacity between actions and states, maximises the influence of an agent on its near future. It has been shown to be a good model of biological behaviour in the absence of an extrinsic goal. But empowerment is also prohibitively hard to compute, especially in nonlinear continuous spaces. We introduce an efficient, amortised method for learning empowerment-maximising policies. We demonstrate that our algorithm can reliably handle continuous dynamical systems using system dynamics learned from raw data. The resulting policies consistently drive the agents into states where they can use their full potential.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Yoshua Bengio
- Pieter Abbeel
- Mohit Bansal
- Jianfeng Gao
- Ruslan Salakhutdinov
- Trevor Darrell
- Chelsea Finn
Citations in the corpus
- Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “This is generally achieved by defining a heuristic based on some measurement of surprise, information gain, compression, or empowerment (Schmidhuber, 1991; Klyubin et al., 2005; Kaplan and Oudeyer, 2006; Schmidhuber, 2008; Frank et al., 2014; Mohamed and Rezende, 2015; Bellemare et al., 2016; Houtho”
- Action and Perception as Divergence Minimization (2020, methodology): “, 2013), for image inputs (Mohamed and Rezende, 2015), optimized using a variational bound (Karl et al., 2017), combined with input density exploration (de Abril and Kanai, 2018) and task rewards (Leibfried et al.”
- Learning Dynamics Model in Reinforcement Learning by Incorporating the Long Term Future (2019, background): “Karl et al. (2017) provide a computationally efficient way to estimate a variational lower bound to empowerement .”
- Understanding the origin of information-seeking exploration in probabilistic objectives for control (2021, unknown): “…& Song, 2018; Okada & Taniguchi, 2019; Shyam, Jaśkowski, & Gomez, 2019), variational bounds on empowerment (Gregor, Rezende, & Wierstra, 2016; Karl et al., 2017), divergence between ensembles (Chua et al., 2018; Shyam et al., 2019) and uncertain state bonuses (Bellemare et al., 2016;…”
- Reset-Free Lifelong Learning with Skill-Space Planning (2020, background): “We note that some “empowerment”-style methods (Gregor et al., 2016; Karl et al., 2017; Eysenbach et al., 2018; Sharma et al., 2019) optimize an objective that aims to resemble a notion of stability; however, this metric is hard to estimate accurately in general, so we do not explicitly use it for…”
- New And Surprising Ways to Be Mean (2018, unknown): “In recent years, several approximations for the maximisation of mutual information, underlying empowerment, have been proposed, drawing on variational inference and deep neural networks [16], [17], [18].”