Learning to Play with Intrinsically-Motivated Self-Aware Agents

By Nick Haber, Damian Mrowca, Li Fei-Fei, Daniel L. K. Yamins. 2018

126 citations recorded by the source corpus.

Infants are experts at playing, with an amazing ability to generate novel structured behaviors in unstructured environments that lack clear extrinsic reward signals. We seek to mathematically formalize these abilities using a neural network that implements curiosity-driven intrinsic motivation. Using a simple but ecologically naturalistic simulated environment in which an agent can move and interact with objects it sees, we propose a "world-model" network that learns to predict the dynamic consequences of the agent's actions. Simultaneously, we train a separate explicit "self-model" that allows the agent to track the error map of its own world-model, and then uses the self-model to adversarially challenge the developing world-model. We demonstrate that this policy causes the agent to explore novel and informative interactions with its environment, leading to the generation of a spectrum of complex behaviors, including ego-motion prediction, object attention, and object gathering. Moreover, the world-model that the agent learns supports improved performance on object dynamics prediction, detection, localization and recognition tasks. Taken together, our results are initial steps toward creating flexible autonomous agents that self-supervise in complex novel physical environments.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

On the Binding Problem in Artificial Neural Networks (2020, background): “The ability to interact with the environment may additionally enable an (embodied) agent to autonomously acquire multi-exemplar data for a particular relation (Schmidhuber, 2015; Haber et al., 2018).”
COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Exploration (2019, methodology): “We also considered using the action space of Haber et al. (2018), which has a distinct component controlling each object in the scene (according to an arbitrary ordering).”
Flexible Neural Representation for Physics Prediction (2018, methodology): “However, future predictors operating directly on 2D image representations often fail to generate sharp object boundaries and struggle with occlusions and remembering objects when they are no longer visually observable [1, 17, 16, 28, 29, 33, 34, 19].”
Exploration by Random Network Distillation (2018, background): “Generic examples of such problems include forward dynamics (Schmidhuber, 1991b; Stadie et al., 2015; Achiam & Sastry, 2017; Pathak et al., 2017; Burda et al., 2018) and inverse dynamics (Haber et al., 2018).”
Emergent Tool Use From Multi-Agent Autocurricula (2019, background): “[49] Nick Haber, Damian Mrowca, Stephanie Wang, Li F Fei-Fei, and Daniel L Yamins.”
Automated curricula through setter-solver interactions (2019, background): “Exploration Some work leverages exploration to encourage state diversity (Gregor et al., 2016b), state-transition surprise (Burda et al., 2018; Haber et al., 2018), or distinguishable skills (Eysenbach et al., 2018).”
World Discovery Models (2019, unknown): “Prediction Error (PE) (Haber et al., 2018; Achiam & Sastry, 2017): The PE model uses the same architecture and the same losses than NDIGO.”
Never Give Up: Learning Directed Exploration Strategies (2020, methodology): “These intrinsic rewards are proportional to some notion of saliency quantifying how different the current state is from those already visited (Bellemare et al., 2016; Haber et al., 2018; Houthooft et al., 2016; Oh et al., 2015; Ostrovski et al., 2017; Pathak et al., 2017; Stadie et al., 2015).”
Planning to Explore via Self-Supervised World Models (2020, background): “Practical examples of such approaches focus on maximizing prediction error as the intrinsic reinforcement learning objective (Pathak et al., 2017; Burda et al., 2019; Haber et al., 2018).”
Learning to Control PDEs with Differentiable Physics (2020, background): “It is typically of vital importance to not only understand the unperturbed physical behavior but also anticipate how the environment reacts to an agent interacting with it [15, 7].”
An Exploration of Embodied Visual Exploration (2020, methodology): “These metrics quantify the extent to which the agent visits things of interest such as area [14, 56, 13, 18], objects [18, 23], and landmarks.”
Incremental Object Learning From Contiguous Views (2019, unknown): “[18] shares our interest in play behavior.”
Shaping embodied agent behavior with activity-context priors from egocentric video (2021, background): “For grasping, prior work studies curiosity [26] and disagreement [42] as intrinsic motivation.”
An Introduction to Deep Reinforcement Learning (2018, unknown): “In (Pathak et al., 2017; Haber et al., 2018), the agent optimizes both a model of its environment and a separate model that predicts the error/uncertainty of its own model.”