Meta reinforcement learning as task inference

By Jan Humplik, Alexandre Galashov, Leonard Hasenclever, Pedro A. Ortega, Yee Whye Teh, Nicolas Manfred Otto Heess. 2019

142 citations recorded by the source corpus.

Humans achieve efficient learning by relying on prior knowledge about the structure of naturally occurring tasks. There is considerable interest in designing reinforcement learning (RL) algorithms with similar properties. This includes proposals to learn the learning algorithm itself, an idea also known as meta learning. One formal interpretation of this idea is as a partially observable multi-task RL problem in which task information is hidden from the agent. Such unknown task problems can be reduced to Markov decision processes (MDPs) by augmenting an agent's observations with an estimate of the belief about the task based on past experience. However estimating the belief state is intractable in most partially-observed MDPs. We propose a method that separately learns the policy and the task belief by taking advantage of various kinds of privileged information. Our approach can be very effective at solving standard meta-RL environments, as well as a complex continuous control environment with sparse rewards and requiring long-term memory.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Solving Rubik's Cube with a Robot Hand (2019, background): “A common theme in research is to try to exploit a shared structure in a distribution of environments, to quickly identify and adapt to previously unseen cases [55, 94, 47, 54] There are works that directly treat meta-learning as identification of a dynamics model of the environment [19] while others”
Stabilizing Transformers for Reinforcement Learning (2019, methodology): “We next demonstrate that the GTrXL scales better compared to an LSTM when an environment’s temporal horizon is increased, using the “Numpad” continuous control task of (Humplik et al., 2019) which allows an easy combinatorial increase in the temporal horizon.”
Tell me why! Explanations support learning relational and causal structure (2021, background): “More broadly, supervising task inference can improve meta-learning (Rakelly et al., 2019; Humplik et al., 2019).”
Model-based Adversarial Meta-Reinforcement Learning (2020, background): “Some recent meta-RL algorithms propose to condition the policy on a latent representation of the task [16, 21, 38, 53, 59].”
VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning (2019, unknown): “Closely related to our approach is the recent work of Humplik et al. (2019).”
Fast Adaptation to New Environments via Policy-Dynamics Value Functions (2020, background): “…for fast adaptation to new environments is meta reinforcement learning (meta RL) (Cully et al., 2015; Finn et al., 2017; Wang et al., 2017; Duan et al., 2016; Xu et al., 2018; Houthooft et al., 2018; Sæmundsson et al., 2018; Nagabandi et al., 2018; Humplik et al., 2019; Rakelly et al., 2019).”
Skill-based Meta-Reinforcement Learning (2022, background): “…al., 2018; Nagabandi et al., 2019; Clavera et al., 2019; 2018; Rakelly et al., 2019; Vuorio et al., 2019; Yang et al., 2019; Zintgraf et al., 2019; Humplik et al., 2019; Zintgraf et al., 2020; Liu et al., 2021) hold the promise of allowing learning agents to quickly adapt to novel tasks by…”
Offline Meta Reinforcement Learning - Identifiability Challenges and Effective Data Collection Strategies (2021, methodology): “Technically, the VariBAD algorithm in [36] is limited to on-policy RL, and the off-policy method in [18] requires specific task descriptors during learning, while VariBAD, which our work is based on, does not.”
Tell me why! - Explanations support learning of relational and causal structure (2021, background): “More broadly, supervising task inference can improve meta-learning (Rakelly et al., 2019; Humplik et al., 2019).”
Offline Meta-Reinforcement Learning with Advantage Weighting (2020, methodology): “Contextual meta-RL methods condition a neural network on experience using a recurrent network [34, 4, 5], a recursive network [23], or a stochastic inference network [29, 39, 14, 31].”
Offline Meta Reinforcement Learning (2020, methodology): “Technically, the VariBAD algorithm in [39] is limited to on-policy RL, and the off-policy method in [22] requires specific task descriptors during learning, while VariBAD, which our work is based on, does not [39].”
Block Contextual MDPs for Continual Learning (2021, background): “Other works have proposed modeling meta-RL as task inference Humplik et al. (2019); Kamienny et al. (2020) but these works generally assume access to some privileged information (like task-id ) during training.”
Efficient Transformers in Reinforcement Learning using Actor-Learner Distillation (2021, methodology): “Similar to ALD, various works have looked at exploiting asymmetries between acting and learning in reinforcement learning (Pinto et al., 2017; Humplik et al., 2019).”