Learning to Learn: Meta-Critic Networks for Sample Efficient Learning

By Flood Sung, Li Zhang, Tao A. Xiang, Timothy M. Hospedales, Yongxin Yang. 2017

140 citations recorded by the source corpus.

We propose a novel and flexible approach to meta-learning for learning-to-learn from only a few examples. Our framework is motivated by actor-critic reinforcement learning, but can be applied to both reinforcement and supervised learning. The key idea is to learn a meta-critic: an action-value function neural network that learns to criticise any actor trying to solve any specified task. For supervised learning, this corresponds to the novel idea of a trainable task-parametrised loss generator. This meta-critic approach provides a route to knowledge transfer that can flexibly deal with few-shot and semi-supervised conditions for both reinforcement and supervised learning. Promising results are shown on both reinforcement and supervised learning problems.

Subfields: Artificial Intelligence, Reinforcement Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

In Meta & Continual Learning

Citations in the corpus

Meta-SGD: Learning to Learn Quickly for Few Shot Learning (2017, background): “Such a two-tiered learning to learn strategy for meta-learning has been applied successfully to few-shot learning on classiﬁcation [7, 18, 19, 25], regression [7, 19], and reinforcement learning [6, 7, 17, 23, 26].”
CoMPS: Continual Meta Policy Search (2021, background): “…(Finn et al., 2017; Gupta et al., 2018b; Rothfuss et al., 2018; Clavera et al., 2018; Mendonca et al., 2019; Zintgraf et al., 2019; Mitchell et al., 2020), and training a critic to provide policy gradients (Sung et al., 2017; Houthooft et al., 2018; Yu et al., 2018; Chebotar et al., 2019).”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “Moreover, another group of recent approaches focuses on learning a meta-critic, which explicitly guides updates to the agent’s policy rather than simply guiding its actions (Sung et al., 2017; Xu et al., 2019; Harb et al., 2020).”
Watch, Try, Learn: Meta-Learning from Demonstrations and Reward (2019, neutral): “In contrast to work in multi-task and meta-reinforcement learning [5, 42, 8, 24, 16, 39, 25, 32, 13], our approach learns to use one demonstration to address the meta-exploration problem [11, 36].”
Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables (2019, methodology): “, 2018a), meta-learned loss functions (Sung et al., 2017; Houthooft et al., 2018), or hyperparameters (Xu et al.”
Model-Based Reinforcement Learning via Meta-Policy Optimization (2018, methodology): “Metalearning, in the context of RL, aims to learn a policy that adapts fast to new tasks or environments [11, 12, 13, 14, 15].”
Meta-Learning in Neural Networks: A Survey (2020, background): “This has potential beneﬁts such as leading to a learned loss that is easier to optimize (e.g., less local minima) than commonly used ones [23], [112], [113], leads to faster learning with improved generalization [45], [114], [115], or one whose minima correspond to a model more robust to domain…”
Learning to Adapt in Dynamic, Real-World Environments through Meta-Reinforcement Learning (2018, background): “Meta-learning in the context of reinforcement learning has largely focused on model-free approaches (Duan et al., 2016; Wang et al., 2016; Sung et al., 2017; Al-Shedivat et al., 2017).”
Meta-Reinforcement Learning of Structured Exploration Strategies (2018, background): “Meta-learning algorithms learn optimizers (Andrychowicz et al., 2016), update rules (Ravi & Larochelle, 2017), or entire RL algorithms (Duan et al., 2016b; Wang et al., 2016; Sung et al., 2017).”
Learning to Learn: Hierarchical Meta-Critic Networks (2019, methodology): “For MC and MAML methods, the network architecture and hyper-parameter settings are same as the original paper, which can be found in literature [20] and [34] formore details.”
Unsupervised Meta-Learning for Reinforcement Learning (2018, background): “These approaches have been extended into the setting of RL (Duan et al., 2016b; Wang et al., 2016; Finn et al., 2017a; Sung et al., 2017; Gupta et al., 2018; Mendonca et al., 2019; Houthooft et al., 2018; Stadie et al., 2018; Rakelly et al., 2019; Nagabandi et al., 2018a).”
Online Meta-Critic Learning for Off-Policy Actor-Critic Methods (2020, background): “No costly offline learning on a task family is required as in Houthooft et al. (2018); Sung et al. (2017).”
Guided Meta-Policy Search (2019, unknown): “…et al., 2017a; Houthooft et al., 2018), current meta-RL methods are generally limited to much simpler domains, such as relatively low-dimensional continuous control tasks (Finn et al., 2017a; Sung et al., 2017) and navigation with discrete action commands (Duan et al., 2016; Mishra et al., 2018).”
Meta Learning via Learned Loss (2019, methodology): “1For simple gradient descent: h(θ,∇θL(y, fθ(x)) = θ − ψ∇θL(y, fθ(x)) Closest to our method are the works on evolved policy gradients [11], teacher networks [32], meta-critics [27] and meta-gradient RL [33].”
Assessing Generalization in Deep Reinforcement Learning (2018, background): “…problem and is under active investigation (Wang et al., 2016; Duan et al., 2016b; Rajeswaran et al., 2017a; Pinto et al., 2017; Finn et al., 2017; Kansky et al., 2017; Yu et al., 2017; Sung et al., 2017; Leike et al., 2017; Al-Shedivat et al., 2018; Clavera et al., 2018; Sæmundsson et al., 2018).”
From Seeing to Moving: A Survey on Learning for Visual Indoor Navigation (VIN) (2020, background): “In [Sung et al., 2017], the authors conditioned state action values on task embeddings which are extracted from the agent’s interactions, and as a consequence the state action values can quickly adapt to new tasks by feeding into the embeddings generated from new interactions.”
Unsupervised Curricula for Visual Meta-Reinforcement Learning (2019, unknown): “Unlike these works and other meta-RL works [61, 13, 38, 46, 18, 30, 26, 47, 56, 58], we close the loop to jointly perform task acquisition and meta-learning so as to achieve an automatic curriculum to facilitate joint meta-learning and task-level exploration.”