Metacontrol for Adaptive Imagination-Based Optimization

By Jessica B. Hamrick, Andrew J. Ballard, Razvan Pascanu, Oriol Vinyals, Nicolas Manfred Otto Heess, Peter W. Battaglia. 2017

72 citations recorded by the source corpus.

Many machine learning systems are built to solve the hardest examples of a particular task, which often makes them large and expensive to run---especially with respect to the easier examples, which might require much less computation. For an agent with a limited computational budget, this "one-size-fits-all" approach may result in the agent wasting valuable computation on easy examples, while not spending enough on hard examples. Rather than learning a single, fixed policy for solving all instances of a task, we introduce a metacontroller which learns to optimize a sequence of "imagined" internal simulations over predictive models of the world in order to construct a more informed, and more economical, solution. The metacontroller component is a model-free reinforcement learning agent, which decides both how many iterations of the optimization procedure to run, as well as which model to consult on each iteration. The models (which we call "experts") can be state transition models, action-value functions, or any other mechanism that provides information useful for solving the task, and can be learned on-policy or off-policy in parallel with the metacontroller. When the metacontroller, controller, and experts were trained with "interaction networks" (Battaglia et al., 2016) as expert models, our approach was able to solve a challenging decision-making problem under complex non-linear dynamics. The metacontroller learned to adapt the amount of computation it performed to the difficulty of the task, and learned how to choose which experts to consult by factoring in both their reliability and individual computational resource costs. This allowed the metacontroller to achieve a lower overall cost (task loss plus computational cost) than more traditional fixed policy approaches. These results demonstrate that our approach is a powerful framework for using...

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

The frontier of simulation-based inference (2019, background): “For instance, a question that is occasionally discussed in the context of reinforcement learning (37, 38) or Bayesian optimization (39), but has not yet been applied to the likelihood-free setting, is how to make use of multifidelity simulators offering multiple levels of precision or approximations”
Structured agents for physical construction (2019, methodology): “Using a learnt object-based model was recently put forward as an efficient method for planning in RL (Pascanu et al., 2017; Hamrick et al., 2017) and for predicting physical dynamics (Sanchez-Gonzalez et al.”
MONet: Unsupervised Scene Decomposition and Representation (2019, background): “…networks that rely on hand-crafted object representations have recently achieved remarkable results in a wide range of research areas, including reinforcement learning, physical modeling, and multi-agent control [Battaglia et al., 2018, Wang et al., 2018, Hamrick et al., 2017, Hoshen, 2017].”
Relational inductive biases, deep learning, and graph networks (2018, methodology): “They have been used within both model-free (Wang et al., 2018b) and model-based (Hamrick et al., 2017; Pascanu et al., 2017; Sanchez-Gonzalez et al., 2018) continuous control, for model-free reinforcement learning (Hamrick et al., 2018; Zambaldi et al., 2018), and for more classical approaches to…”
Relational Deep Reinforcement Learning (2018, background): “While not a model in the technical sense, it is possible that the agent learns to exploit this relational architectural prior similarly to how an imagination-based agent’s forward model operates [35, 36, 37].”
Imagination-Augmented Agents for Deep Reinforcement Learning (2017, background): “[51]: they present a neural architecture that queries learned expert models, but focus on meta-control for continuous contextual bandit problems.”
Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learning (2017, methodology): “These include decisions trained with REINFORCE (Denoyer & Gallinari, 2014), (Bengio et al., 2015), (Hamrick et al., 2017), Q Learning (Liu & Deng, 2017), and actor-critic methods (McGill & Perona, 2017).”
Graph networks as learnable physics engines for inference and control (2018, methodology): “Our approach may also be useful within imagination-based planning frameworks (Hamrick et al., 2017; Pascanu et al., 2017), as well as integrated architectures with GN-like policies (Wang et al., 2018).”
Automatically Composing Representation Transformations as a Means for Generalization (2018, background): “Our work is one of a small but growing community [32, 40, 61] linking metareasoning with deep learning.”
Learning model-based planning from scratch (2017, methodology): “’s “imagination-based metacontroller” (IBMC) [9], which learned an adaptive optimization policy for one-shot decisionmaking in contextual bandit problems.”
Relational Forward Models for Multi-Agent Learning (2018, positive): “Here we show that, by embedding RFM modules in RL agents, they can learn to coordinate with one another faster than baseline agents, analogous to imagination-augmented agents in single-agent RL settings (Hamrick et al., 2017; Pascanu et al., 2017; Weber et al., 2017).”
Model-Based Planning in Discrete Action Spaces (2017, methodology): “Other works have used forward models for planning in continous action spaces, such as billiards [9], vehicle navigation [12] or robotics [33, 1, 32, 21].”
Small Sample Learning in Big Data Era (2018, background): “, 2017c), and imagination of realistic environment (Chiappa et al., 2017; Racanière et al., 2017; Gemici et al., 2017; Hamrick et al., 2017) based on simulation-based planning and imagination, which leads to data efficiency improvement and novel concept learning.”
Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluids (2018, background): “They often learn a policy based on interaction networks’ rollouts (Racani ` ere et al., 2017; Hamrick et al., 2017; Pascanu et al., 2017).”
On the link between conscious function and general intelligence in humans and machines (2022, background): “For the most faithful implementation of AST, we believe a “metacontroller”, which adapts where and how much computation should be spent in decision making, will be a key component (Hamrick et al., 2017).”