Meta-Gradient Reinforcement Learning

By Zhongwen Xu, Hado van Hasselt, David Silver. 2018

381 citations recorded by the source corpus.

The goal of reinforcement learning algorithms is to estimate and/or optimise the value function. However, unlike supervised learning, no teacher or oracle is available to provide the true value function. Instead, the majority of reinforcement learning algorithms estimate and/or optimise a proxy for the value function. This proxy is typically based on a sampled and bootstrapped approximation to the true value function, known as a return. The particular choice of return is one of the chief components determining the nature of the algorithm: the rate at which future rewards are discounted; when and how values should be bootstrapped; or even the nature of the rewards themselves. It is well-known that these decisions are crucial to the overall success of RL algorithms. We discuss a gradient-based meta-learning algorithm that is able to adapt the nature of the return, online, whilst interacting and learning from the environment. When applied to 57 games on the Atari 2600 environment over 200 million frames, our algorithm achieved a new state-of-the-art performance.

Subfields: Artificial Intelligence, Reinforcement Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

In Meta & Continual Learning

Citations in the corpus

Off-Policy Actor-Critic with Shared Experience Replay (2019, background): “Note that Xu et al. (2018) have not published per-level scores.”
SNAS: Stochastic Neural Architecture Search (2018, background): “To assign credits to actions both temporally and laterally is an important topic in reinforcement learning (Precup, 2000; Schulman et al., 2015; Tucker et al., 2018; Xu et al., 2018).”
RUDDER: Return Decomposition for Delayed Rewards (2018, background): “ ﬁnite time horizon or absorbing states to avoid discounting of the delayed rewards. Discount factors close to 1 have been conﬁrmed to be suited to long delays by meta-gradient reinforcement learning [125]. Grid World: RUDDER is tested on a grid world with delayed reward at the end of an episode. Th”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “To this end, potential directions aligned with this process of discovery through computation include: self-tuning approaches (Xu et al., 2018a; Zahavy et al., 2020), end-to-end skill discovery (Bacon et al., 2017), discovering RL algorithms (Kirsch, van Steenkiste, and Schmidhuber, 2019; Oh,…”
Acme: A Research Framework for Distributed Reinforcement Learning (2020, background): “, 2017) or meta-learning (Al-Shedivat et al., 2017; Finn et al., 2017; Xu et al., 2018), to name a few examples.”
Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables (2019, methodology): “…methods, gradient-based meta-RL methods learn from aggregated experience using policy gradients (Finn et al., 2017; Stadie et al., 2018; Rothfuss et al., 2018; Xu et al., 2018a), meta-learned loss functions (Sung et al., 2017; Houthooft et al., 2018), or hyperparameters (Xu et al., 2018b).”
Meta-Learning in Neural Networks: A Survey (2020, background): “addressed the single-task setting, where the meta-knowledge such as loss [22], [180], reward [174], [181], hyperparameters [172], [173], or exploration strategy [182] are trained online together with the base policy while learning a single task.”
On Inductive Biases in Deep Reinforcement Learning (2019, methodology): “The meta-gradient algorithm Xu et al. (2018) uses the insight that the updates in Equations (2) and (4) are differentiable functions of hyper-parameters such as the discount.”
Dual Policy Distillation (2020, neutral): “Meta-learning methods also make use of a teacher model to improve the sample efﬁciency [Xu et al. , 2018b; Xu et al. , 2018a; Zha et al. , 2019b].”
Meta-learning curiosity algorithms (2020, methodology): “Most work on meta-RL has focused on learning transferable feature representations or parameter values for quickly adapting to new tasks (Finn et al., 2017; Finn, 2018; Clavera et al., 2019) or improving performance on a single task (Xu et al., 2018; Veeriah et al., 2019).”
Human-Timescale Adaptation in an Open-Ended Task Space (2023, methodology): “Other approaches for meta-RL include meta-gradients (Andrychowicz et al., 2016; Finn et al., 2017; Flennerhag et al., 2022; Xu et al., 2018), which can be efficient but often suffer from instability and myopia (Flennerhag et al.”
ETA Prediction with Graph Neural Networks in Google Maps (2021, methodology): “…) with some update function 𝑓 (eg. a gradient step) additionally parameterized by 𝜂 , and a reference meta-parameter value 𝜂 ′ , MetaGradients computes the update for 𝜂 using the following derivative: where the gradient w.r.t. the model parameter is multiplied by a factor 𝜕𝜃 ′ / 𝜕𝜂 .”
Adaptive Auxiliary Task Weighting for Reinforcement Learning (2019, methodology): “Some recent works can be viewed as instantiations of this method, where the meta-parameters are the parameters of the return function [32] or an intrinsic reward function [33].”
Self-Tuning Deep Reinforcement Learning (2020, methodology): “995 as it was found to improve the performance of IMPALA considerably (Xu et al., 2018).”
Bootstrapped Meta-Learning (2021, methodology): “This presents serious challenges for meta-learning as the policy gradient becomes noisy and volatile due to off-policy estimation (Xu et al., 2018; Zahavy et al., 2020).”
Discovering Reinforcement Learning Algorithms (2020, methodology): “This approach has been applied to meta-learn various forms of algorithmic components such as the discount factor [38], intrinsic rewards [41], auxiliary tasks [33], returns [36], auxiliary policy updates [42], off-policy corrections [39], and update target [37].”
Meta Knowledge Distillation (2022, methodology): “Previous meta-hyperparameter-optimization works (Xu et al., 2018; Li et al., 2017; Baik et al., 2020; Khodak et al., 2019) usually employ the first approach to optimize their hyperparameters.”
Meta Learning via Learned Loss (2019, background): “1For simple gradient descent: h(θ,∇θL(y, fθ(x)) = θ − ψ∇θL(y, fθ(x)) Closest to our method are the works on evolved policy gradients [11], teacher networks [32], meta-critics [27] and meta-gradient RL [33].”
Hindsight Foresight Relabeling for Meta-Reinforcement Learning (2021, methodology): “Gradient-based metalearning methods have also been explored (Finn et al., 2017; Nichol & Schulman, 2018; Xu et al., 2018; Zheng et al., 2020).”