Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environments

By Maruan Al-Shedivat, Trapit Bansal, Yuri Burda, Ilya Sutskever, Igor Mordatch, Pieter Abbeel. 2017

376 citations recorded by the source corpus.

Ability to continuously learn and adapt from limited experience in nonstationary environments is an important milestone on the path towards general intelligence. In this paper, we cast the problem of continuous adaptation into the learning-to-learn framework. We develop a simple gradient-based meta-learning algorithm suitable for adaptation in dynamically changing and adversarial scenarios. Additionally, we design a new multi-agent competitive environment, RoboSumo, and define iterated adaptation games for testing various aspects of continuous adaptation strategies. We demonstrate that meta-learning enables significantly more efficient adaptation than reactive baselines in the few-shot regime. Our experiments with a population of agents that learn and compete suggest that meta-learners are the fittest.

Subfields: Artificial Intelligence, Reinforcement Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

In Meta & Continual Learning

Citations in the corpus

Adaptive Gradient-Based Meta-Learning Methods (2019, methodology): “(2017) and has been widely used in practice (Li et al., 2017; Al-Shedivat et al., 2018; Nichol et al., 2018; Jerfel et al., 2018).”
Meta-Gradient Reinforcement Learning (2018, background): “icy can generalise into new tasks faster than learning the policy from scratch. Model-Agnostic Meta-Learning (MAML) [Finn et al., 2017a, Finn and Levine, 2018, Finn et al., 2017b, Grant et al., 2018, Al-Shedivat et al., 2018] learns a good initialisation of the model that can adapt quickly to other ”
Deep Reinforcement Learning for Autonomous Driving: A Survey (2020, methodology): “Continuous adaptation in dynamically changing and adversarial scenarios is presented in [135] via a simple gradient-based meta-learning algorithm.”
Policy Consolidation for Continual Reinforcement Learning (2019, methodology): “To encourage faster learning, as in (Al-Shedivat et al., 2018) and (Bansal et al., 2018), we also trained all agents using a dense reward curriculum in the initial stages of training.”
RMM: Reinforced Memory Management for Class-Incremental Learning (2023, background): “This matches the conclusion of continuous adaptation in [2].”
CoMPS: Continual Meta Policy Search (2021, methodology): “Adaptation without task boundaries or inside of an RL episode has also become a new area of investigation for meta-learning (Nagabandi et al., 2019; Javed & White, 2019; Harrison et al., 2019; Al-Shedivat et al., 2018; He et al., 2019; Harrison et al., 2019).”
Continuous Coordination As a Realistic Scenario for Lifelong Learning (2021, background): “Al-Shedivat et al. (2017) introduced RoboSumo — a 3D environment based on MuJoCo physics simulator that allows pairs of agents to compete against each other.”
Continual Lifelong Learning in Natural Language Processing: A Survey (2020, methodology): “As meta-learning is able to efficiently learn with limited samples, hence such a CL model would adapt quicker in dynamic environments (e.g. Ritter et al., 2018; Al-Shedivat et al., 2018).”
Orthogonal Gradient Descent for Continual Learning (2019, methodology): “Meta learning algorithms use a data-driven inductive bias to enhance learning new tasks (Jerfel et al., 2019; He and Jaeger, 2018; Vuorio et al., 2018; Al-Shedivat et al., 2017; Riemer et al., 2018).”
Meta-Learning Representations for Continual Learning (2019, methodology): “Some meta-learning algorithms have similarly considered optimizations through multiple steps of updating for the few-shot learning setting (Finn et al., 2017; Li et al., 2017; Al-Shedivat et al., 2018; Nagabandi et al., 2019) for learning model initializations.”
Continual Reinforcement Learning in 3D Non-stationary Environments (2019, background): “This is why recent reinforcement learning algorithms for continual learning have been tested only on arguably simple tasks of low/medium input space dimension and complexity [21, 1, 35].”
Functional Regularisation for Continual Learning using Gaussian Processes (2019, neutral): “This is also reminiscent to recent work on Meta-Learning of non-stationary task distributions (Al-Shedivat et al., 2017).”
Learning to Learn without Forgetting By Maximizing Transfer and Minimizing Interference (2018, background): “…(Santoro et al., 2016; Vinyals et al., 2016; Ravi & Larochelle, 2016; Finn et al., 2017) and reinforcement learning across successive tasks (Al-Shedivat et al., 2018), we are not only trying to improve the speed of learning on new data, but also trying to do it in a way that preserves…”
Scalable Recollections for Continual Lifelong Learning (2017, background): “Some approaches to lifelong learning assume that a task is a set of examples chosen from the same distribution (Rusu et al., 2016; Fernando et al., 2017; Shin et al., 2017a; Ramapuram et al., 2017; Al-Shedivat et al., 2017; Lee et al., 2018).”
Acme: A Research Framework for Distributed Reinforcement Learning (2020, background): “, 2017) or meta-learning (Al-Shedivat et al., 2017; Finn et al., 2017; Xu et al., 2018), to name a few examples.”
Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RL (2018, background): “Other meta-learning works have considered non-stationarity within a task (Al-Shedivat et al., 2017) and episodes involving multiple tasks at meta-test time (Ritter et al., 2018), but they do not consider continual online adaptation with unknown task separation.”
Online Meta-Learning (2019, neutral): “While some meta-learning works have considered online learning settings at meta-test time (Santoro et al., 2016; Al-Shedivat et al., 2017; Nagabandi et al., 2018), nearly all prior meta-learning algorithms assume that the meta-training tasks come from a stationary distribution.”