Meta Learning via Learned Loss

By Yevgen Chebotar, Artem Molchanov, Sarah Bechtle, Ludovic Righetti, F. Meier, G. Sukhatme. 2019

126 citations recorded by the source corpus.

Typically, loss functions, regularization mechanisms and other important aspects of training parametric models are chosen heuristically from a limited set of options. In this paper, we take the first step towards automating this process, with the view of producing models which train faster and more robustly. Concretely, we present a meta-learning method for learning parametric loss functions that can generalize across different tasks and model architectures. We develop a pipeline for “meta-training” such loss functions, targeted at maximizing the performance of the model trained under them. The loss landscape produced by our learned losses significantly improves upon the original task-specific losses in both supervised and reinforcement learning tasks. Furthermore, we show that our meta-learning framework is flexible enough to incorporate additional information at meta-train time. This information shapes the learned loss function such that the environment does not need to provide this information during meta-test time. We make our code available at https://sites.google.com/view/mlthree

Subfields: Artificial Intelligence, Software

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Software

Citations in the corpus

Meta-Learning in Neural Networks: A Survey (2020, background): “This can lead to a learned loss that is easier to optimize (less local minima) than standard alternatives [22], [25], [117], provides faster learning with improved generalization [43], [118], [119], [120], robustness to label noise [121], or whose minima corresponds to a model more robust to domain ”
Meta-Gradient Reinforcement Learning with an Objective Discovered Online (2020, unknown): “Algorithm Algorithm properties What is meta-learned? IDBD, SMD [31, 28] → learning rate SGD(2) [1] ← optimiser RL(2), Meta-RL [9, 40] X recurrent network MAML, REPTILE [11, 24] ← initial params Meta-Gradient [44, 47] → γ, λ, reward Meta-Gradient [39, 45, 41] ← auxiliary tasks, hype”
Evolving Reinforcement Learning Algorithms (2021, background): “Learning RL algorithms or their components, such as a reward bonus or value update function, has been studied previously with meta-gradients (Kirsch et al., 2020; Chebotar et al., 2019; Oh et al., 2020), evolutionary strategies (Houthooft et al., 2018), and RNNs (Duan et al., 2016).”
Generalized Inner Loop Meta-Learning (2019, methodology): “Finally, recent work by Bechtle et al. (2019) has introduced the ML 3 framework for learning un-constrained loss functions as parametric models, through exploiting second-order gradients of a meta-loss with regard to the parameters of the inner loss.”
Offline Meta-Reinforcement Learning with Advantage Weighting (2020, methodology): “…into the meta-level optimization (Finn et al., 2017a; Nagabandi et al., 2019; Rothfuss et al., 2018; Zintgraf et al., 2019; Gupta et al., 2018; Mendonca et al., 2019; Yang et al., 2019), potentially using a learned loss function (Houthooft et al., 2018; Bechtle et al., 2019; Kirsch et al., 2020).”
Introducing Symmetries to Black Box Meta Reinforcement Learning (2021, background): “2020), meta-learning hyperparameters (Sutton 1992; Xu, van Hasselt, and Silver 2018) or meta-learning objective functions corresponding to the learning algorithm (Houthooft et al. 2018; Kirsch, van Steenkiste, and Schmidhuber 2020; Xu et al. 2020; Oh et al. 2020; Bechtle et al. 2021).”
Adaptive Risk Minimization: A Meta-Learning Approach for Tackling Group Shift (2020, methodology): “Our method can also likely be further improved with more sophisticated meta-learning approaches, such as optimization based meta-learning using a learned loss function [2, 4], and this is another exciting direction for future work.”
Online Meta-Critic Learning for Off-Policy Actor-Critic Methods (2020, methodology): “A few studies learn hyperparameters [40, 42], loss functions [3, 15, 36] or rewards [44] that steer the learning of standard optimisers.”
What Can Learned Intrinsic Rewards Capture? (2019, background): “In particular, our multi-lifetime formulation is similar to ML 3 (Bechtle et al., 2019).”
Discovering Reinforcement Learning Algorithms (2020, methodology): “4 Balancing Agent Hyperparameters for Stabilisation (α) While previous approaches [6, 17] used fixed agent hyperparameters (e.”
Improving Generalization in Meta Reinforcement Learning using Learned Objectives (2019, background): “Most recently, Bechtle et al. (2019) introduced learned loss functions for reinforcement learning that also make use of second-order gradients, but use a policy gradient estimator instead of a Q-function.”