Meta-Q-Learning

By Rasool Fakoor, Pratik Chaudhari, Stefano Soatto, Alex Smola. 2019

167 citations recorded by the source corpus.

This paper introduces Meta-Q-Learning (MQL), a new off-policy algorithm for meta-Reinforcement Learning (meta-RL). MQL builds upon three simple ideas. First, we show that Q-learning is competitive with state-of-the-art meta-RL algorithms if given access to a context variable that is a representation of the past trajectory. Second, a multi-task objective to maximize the average reward across the training tasks is an effective method to meta-train RL policies. Third, past data from the meta-training replay buffer can be recycled to adapt the policy on a new task using off-policy updates. MQL draws upon ideas in propensity estimation to do so and thereby amplifies the amount of available data for adaptation. Experiments on standard continuous-control benchmarks suggest that MQL compares favorably with the state of the art in meta-RL.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Meta-Learning in Neural Networks: A Survey (2020, background): “Initial Condition [16], [79], [88], [102], [166], [166]–[168] [169]–[171] [16], [63], [64] [172], [173]”
Hindsight Foresight Relabeling for Meta-Reinforcement Learning (2021, background): “Recent works [6, 7] have sought to 29 incorporate off-policy RL [8, 9] into meta-RL to improve sample efficiency.”
Offline Meta Reinforcement Learning - Identifiability Challenges and Effective Data Collection Strategies (2021, background): “This, however, requires some care, as Q-learning applies to states of fully observed systems.”
Towards Effective Context for Meta-Reinforcement Learning: an Approach based on Contrastive Learning (2020, methodology): “To alleviate this problem, a number of methods (Rakelly et al., 2019; Fakoor et al., 2020) are proposed to meta-learn a policy that is able to adapt with off-policy data by leveraging context information.”
Offline Meta Reinforcement Learning (2020, methodology): “This, however, requires some care, as Q-learning applies to states of fully observed systems.”
Parrot: Data-Driven Behavioral Priors for Reinforcement Learning (2020, methodology): “…(Duan et al., 2016; Wang et al., 2016; Finn et al., 2017a; Mishra et al., 2017; Rakelly et al., 2019; Mendonca et al., 2019; Zintgraf et al., 2020; Fakoor et al., 2020) and meta-imitation methods (Duan et al., 2017; Finn et al., 2017c; Huang et al., 2018; James et al., 2018; Paine et al., 2018;…”
Model-based Adversarial Meta-Reinforcement Learning (2020, methodology): “Fakoor et al. (2019) proposes ways to reuse data from the meta-training phase during meta-adaptation by employing propensity score estimation.”
AdaRL: What, Where, and How to Adapt in Transfer Reinforcement Learning (2021, methodology): “In the POMDP setting, we compare with PNN (Rusu et al., 2016), PSM (Agarwal et al., 2021a) and MQL (Fakoor et al., 2020).”
Improving Generalization in Meta-RL with Imaginary Tasks from Latent Dynamics Mixture (2021, methodology): “For MuJoCo tasks we report the results of PEARL [35], MQL [8], and MIER [30] with number of rollouts per iteration ( N ) equal to 3, 11 and 3 respectively.”
Offline Meta-Reinforcement Learning with Advantage Weighting (2020, methodology): “Contextual meta-RL methods condition a neural network on experience using a recurrent network (Wang et al., 2016; Duan et al., 2016; Fakoor et al., 2020), a recursive network (Mishra et al., 2017), or a stochastic inference net- work (Rakelly et al., 2019; Zintgraf et al., 2020; Humplik et al.,…”