Meta Reinforcement Learning with Latent Variable Gaussian Processes

By Steindór Sæmundsson, Katja Hofmann, Marc Peter Deisenroth. 2018

148 citations recorded by the source corpus.

Learning from small data sets is critical in many practical applications where data collection is time consuming or expensive, e.g., robotics, animal experiments or drug design. Meta learning is one way to increase the data efficiency of learning algorithms by generalizing learned concepts from a set of training tasks to unseen, but related, tasks. Often, this relationship between tasks is hard coded or relies in some other way on human expertise. In this paper, we frame meta learning as a hierarchical latent variable model and infer the relationship between tasks automatically from data. We apply our framework in a model-based reinforcement learning setting and show that our meta-learning model effectively generalizes to novel tasks by identifying how new tasks relate to prior ones from minimal data. This results in up to a 60% reduction in the average interaction time needed to solve tasks compared to strong baselines.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Control and Systems Engineering, Reinforcement Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Control and Systems Engineering

In Reinforcement Learning

In Meta & Continual Learning

Citations in the corpus

CoMPS: Continual Meta Policy Search (2021, methodology): “…black box models (Duan et al., 2016; Wang et al., 2016; Mishra et al., 2017; Stadie et al., 2018; Rakelly et al., 2019; Zintgraf et al., 2020; Sæmundsson et al., 2018; Fakoor et al., 2019; Sun et al., 2019; Dorfman & Tamar, 2020), applying gradient descent from initial parameters (Finn et…”
Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning (2019, methodology): “n [18,19,20], continuous control domains with parametric variation across tasks [10,21,22,23], bandit problems [19,18,20,24], levels of an arcade game [25], and locomotion tasks with varying dynamics [26,27]. Complementary to these evaluations, we aim to develop a testbed of tasks and an evaluation ”
Watch, Try, Learn: Meta-Learning from Demonstrations and Reward (2019, neutral): “In contrast to work in multi-task and meta-reinforcement learning [5, 42, 8, 24, 16, 39, 25, 32, 13], our approach learns to use one demonstration to address the meta-exploration problem [11, 36].”
Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables (2019, methodology): “Recently, meta-RL methods have been developed for metalearning dynamics models (Nagabandi et al., 2019; Sæmundsson et al., 2018) and policies (Finn et al.”
Learning to Adapt in Dynamic, Real-World Environments through Meta-Reinforcement Learning (2018, methodology): “In prior gradient-based and model-based meta-RL approaches (Finn et al., 2017; Sæmundsson et al., 2018), the M has corresponded to M trajectories, leading to episodic adaptation.”
Guided Meta-Policy Search (2019, unknown): “…from a learned initialization (Finn et al., 2017a; Gupta et al., 2018; Rothfuss et al., 2018), using a learned critic that provides gradients to the policy (Sung et al., 2017; Houthooft et al., 2018), or using a planner and an adaptable model (Clav-era et al., 2018; Sæmundsson et al., 2018).”
Generalized Hidden Parameter MDPs Transferable Model-based RL in a Handful of Trials (2020, background): “Previous work showed that for simple, low-dimensional systems, agents can infer an effective representation of the hidden parameters, and generalize the dynamics to novel parameter settings (Killian et al. 2017; Sæmundsson, Hofmann, and Deisenroth 2018).”
Offline Meta-Reinforcement Learning with Advantage Weighting (2020, methodology): “Contextual meta-RL methods condition a neural network on experience using a recurrent network [34, 4, 5], a recursive network [23], or a stochastic inference network [29, 39, 14, 31].”
NoRML: No-Reward Meta Learning (2019, unknown): “Prior model-based RL approaches have considered the problem of learning to adapt to different dynamics through meta-RL [4, 24] or through learned priors [11].”
Meta-Inverse Reinforcement Learning with Probabilistic Context Variables (2019, methodology): “While prior multi-task and meta-RL methods [15, 26, 29] have investigated the effectiveness of applying latent variable generative models to learning task embeddings, we focus on the IRL problem instead.”
Causal Induction from Visual Observations for Goal Directed Tasks (2019, unknown): “…been previously studied in in the context of domain adaptation (Tzeng et al., 2015; Peng et al., 2017), system identiﬁcation (Yu et al., 2017; Zhou et al., 2019), meta-learning (Finn et al., 2017; Sæmundsson et al., 2018), and multi-task learning (Andrychowicz et al., 2017; Schaul et al., 2015).”
Model-based Adversarial Meta-Reinforcement Learning (2020, background): “Some algorithms [22, 31, 32, 40] also propose to share a dynamical model across tasks during meta-training and perform model-based adaptation in new tasks.”
Mathematics for Machine Learning (2020, background): “Examples, where such latent variables are helpful, are principal component analysis for dimensionality reduction (Chapter 10), Gaussian mixture models for density estimation (Chapter 11), hidden Markov models (Maybeck, 1979) or dynamical systems (Ghahramani and Roweis, 1999; Ljung, 1999) for time-se”
Assessing Generalization in Deep Reinforcement Learning (2018, methodology): “…problem and is under active investigation (Wang et al., 2016; Duan et al., 2016b; Rajeswaran et al., 2017a; Pinto et al., 2017; Finn et al., 2017; Kansky et al., 2017; Yu et al., 2017; Sung et al., 2017; Leike et al., 2017; Al-Shedivat et al., 2018; Clavera et al., 2018; Sæmundsson et al., 2018).”
Task-Agnostic Online Reinforcement Learning with an Infinite Mixture of Gaussian Processes (2020, methodology): “Meta-learning algorithms in general consist of a base (quick) learner and a meta (slow) learner [18, 19] and have recently been combined with RL to solve nonstationary problems via model-free [20, 21, 22] and model-based approaches [3, 2].”
CAML: Fast Context Adaptation via Meta-Learning (2018, background): “Other meta-learning methods are also motivated by the fact that learning in a high-dimensional parameter space can pose practical difﬁculties, and fast adaptation in lower dimensional space can be better (e.g., Sæmundsson et al. (2018); Zhou et al. (2018); Rusu et al. (2018)).”
ProMP: Proximal Meta-Policy Search (2018, background): “, 2018a), model-based RL (Clavera et al., 2018; Saemundsson et al., 2018), learning in competitive environments (Al-Shedivat et al.”