Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RL

By Anusha Nagabandi, Chelsea Finn, Sergey Levine. 2018

215 citations recorded by the source corpus.

Humans and animals can learn complex predictive models that allow them to accurately and reliably reason about real-world phenomena, and they can adapt such models extremely quickly in the face of unexpected changes. Deep neural network models allow us to represent very complex functions, but lack this capacity for rapid online adaptation. The goal in this paper is to develop a method for continual online learning from an incoming stream of data, using deep neural network models. We formulate an online learning procedure that uses stochastic gradient descent to update model parameters, and an expectation maximization algorithm with a Chinese restaurant process prior to develop and maintain a mixture of models to handle non-stationary task distributions. This allows for all models to be adapted as necessary, with new models instantiated for task changes and old models recalled when previously seen tasks are encountered again. Furthermore, we observe that meta-learning can be used to meta-train a model such that this direct online adaptation with SGD is effective, which is otherwise not the case for large function approximators. In this work, we apply our meta-learning for online learning (MOLe) approach to model-based reinforcement learning, where adapting the predictive model is critical for control; we demonstrate that MOLe outperforms alternative prior methods, and enables effective continuous adaptation in non-stationary task distributions such as varying terrains, motor failures, and unexpected disturbances.

Subfields: Artificial Intelligence, Reinforcement Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

In Meta & Continual Learning

Citations in the corpus

CoMPS: Continual Meta Policy Search (2021, methodology): “Adaptation without task boundaries or inside of an RL episode has also become a new area of investigation for meta-learning (Nagabandi et al., 2019; Javed & White, 2019; Harrison et al., 2019; Al-Shedivat et al., 2018; He et al., 2019; Harrison et al., 2019).”
Online Continual Learning in Image Classification: An Empirical Survey (2021, background): “Although CL is also studied in reinforcement learning [19, 20, 21] and more recently in unsupervised learning [22], the image classification problem is still the main focus for many CL researchers.”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “…Kim, Yoon, Dia, Kim, Bengio, and Ahn, 2018; Rothfuss, Lee, Clavera, Asfour, and Abbeel, 2018; Flennerhag, Moreno, Lawrence, and Damianou, 2018; Nagabandi, Finn, and Levine, 2018; Mendonca, Gupta, Kralev, Abbeel, Levine, and Finn, 2019; Finn, Rajeswaran, Kakade, and Levine, 2019; Lin, Thomas,…”
Lifelong Policy Gradient Learning of Factored Policies for Faster Training Without Forgetting (2020, background): “A similar approach has been explored in the context of model-based RL [25], but their focus was discovering when new tasks were encountered in the absence of task indicators.”
A Neural Dirichlet Process Mixture Model for Task-Free Continual Learning (2020, methodology): “Nagabandi et al. (2019) apply DPM to the model-based reinforcement learning to predict the next state from a given state-action pair.”
Adaptive Online Planning for Continual Lifelong Learning (2019, background): “[23] learn multiple models to represent different tasks, which contrasts with our single unified networks.”
Meta-Learning Representations for Continual Learning (2019, methodology): “Some meta-learning algorithms have similarly considered optimizations through multiple steps of updating for the few-shot learning setting (Finn et al., 2017; Li et al., 2017; Al-Shedivat et al., 2018; Nagabandi et al., 2019) for learning model initializations.”
Task Agnostic Continual Learning via Meta Learning (2019, methodology): “We consider an online learning scenario similar to Hochre-iter et al. (2001); Nagabandi et al. (2019), where at each time step t , a model ˆ f parametrised by θ t receives an observation x t and makes a prediction ˆ y t := ˆ f ( x t ; θ t ) .”
Online Meta-Learning (2019, neutral): “While some meta-learning works have considered online learning settings at meta-test time (Santoro et al., 2016; Al-Shedivat et al., 2017; Nagabandi et al., 2018), nearly all prior meta-learning algorithms assume that the meta-training tasks come from a stationary distribution.”
Challenges of Real-World Reinforcement Learning (2019, background): “System iden-tiﬁcation involves training a policy that can determine on-line the environment in which it is operating and modify the policy accordingly (Finn et al., 2017; Nagabandi et al., 2018).”
Lifelong Learning of Compositional Structures (2020, background): “To tackle this problem, some approaches focus on lifelong learning of factored or clustered models [27, 20].”
Generalizing from a Few Examples (2019, background): “onment starts to draw attention [28, 32, 77]. Existing application includes one-shot imitation [28], multi-armed bandits [28], visual navigation [28], continuous control in locomotion [32]. Recently, [2, 82] further considers these applications in dynamic environment. 6.4 Theory In FSL problem, reac”
Continuous Meta-Learning without Tasks (2019, methodology): “Nagabandi et al. (2019b) and Jerfel et al. (2019) aim to detect task changes via combining mean estimation of the dependent variable with MAML models.”
Variable-Shot Adaptation for Online Meta-Learning (2020, background): “Finally, our work considers a problem setting that is distinct from works that a study online or continual learning in the inner loop of meta-learning (Al-Shedivat et al., 2017; Nagabandi et al., 2018; Javed & White, 2019; Denevi et al., 2019; He et al., 2019; Harrison et al., 2019).”
TaskNorm: Rethinking Batch Normalization for Meta-Learning (2020, background): “In particular, meta-learning systems have been demonstrated to excel at complex learning tasks such as few-shot (Snell et al., 2017; Finn et al., 2017) and continual learning (Nagabandi et al., 2019; Requeima et al., 2019a; Jerfel et al., 2019).”
How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Composition (2022, methodology): “A similar approach in the context of model-based RL represents the dynamics of the tasks via an aggregation of supervised models (Nagabandi et al., 2019).”