Hierarchically Structured Meta-learning

By Huaxiu Yao, Ying Wei, Junzhou Huang, Zhenhui Jessie Li. 2019

223 citations recorded by the source corpus.

In order to learn quickly with few samples, meta-learning utilizes prior knowledge learned from previous tasks. However, a critical challenge in meta-learning is task uncertainty and heterogeneity, which can not be handled via globally sharing knowledge among tasks. In this paper, based on gradient-based meta-learning, we propose a hierarchically structured meta-learning (HSML) algorithm that explicitly tailors the transferable knowledge to different clusters of tasks. Inspired by the way human beings organize knowledge, we resort to a hierarchical task clustering structure to cluster tasks. As a result, the proposed approach not only addresses the challenge via the knowledge customization to different clusters of tasks, but also preserves knowledge generalization among a cluster of similar tasks. To tackle the changing of task relationship, in addition, we extend the hierarchical structure to a continual learning environment. The experimental results show that our approach can achieve state-of-the-art performance in both toy-regression and few-shot image classification problems.

Subfields: Artificial Intelligence, Cancer Research, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cancer Research

In Computer Vision and Pattern Recognition

Citations in the corpus

Meta-Learning in Neural Networks: A Survey (2020, background): “The most common application of meta-learning is few-shot multi-class image recognition, where the inner and outer loss functions are typically the cross entropy over training and validation data respectively [14], [23], [75], [77], [78], [88], [90], [98], [99], [102], [105], [201], [202], [203], [20”
Online Structured Meta-learning (2020, methodology): “First, all online meta-learning methods (i.e., FTML, DPM, HSML, OSML) achieves better performance than the non-meta-learning ones (i.e., NT, FT), which further demonstrate the effectiveness of task-speciﬁc adaptation.”
Few-Shot Learning With a Strong Teacher (2021, methodology): “Based on the meta-trained model initialization [7], [58], [59], [60], [61], [62], the few-shot learner can adapt it to a good classiﬁer for each few-shot task with a small number of gradient descent steps.”
Bayesian Meta Sampling for Fast Uncertainty Adaptation (2020, background): “…2017), the Reptile model (Nichol et al., 2018), and the lately extensions to an online setting (Finn et al., 2019), to model hierarchical relation (Yao et al., 2019) and sequential strategies (Ortega et al., 2019), and to its stable version Antoniou et al. (2019) and to some theoretical analysis…”
Meta-learning on Heterogeneous Information Networks for Cold-start Recommendation (2020, methodology): “2 Meta-learning for Recommendation Our work is inspired by the optimization-based meta-learning [9, 32], which optimizes globally shared parameters (i.”
Meta-Learning with Neural Tangent Kernels (2021, unknown): “…(MAML) (Finn et al., 2017) is perhaps one of the most popular and ﬂexible ones, with a number of follow-up works such as (Nichol et al., 2018; Finn et al., 2018; Yao et al., 2019; Khodak et al., 2019a;b; Denevi et al., 2019; Fallah et al., 2020; Lee et al., 2020; Tripuraneni et al., 2020).”
How to Train Your MAML to Excel in Few-Shot Classification (2021, background): “Another direction is to enable task-specific initialization [75, 99, 109, 111], which often needs additional sub-networks.”
Graph Few-shot Learning via Knowledge Transfer (2019, background): “…parameters that can be updated by a few gradient steps in future tasks (Finn, Abbeel, and Levine 2017; Finn, Xu, and Levine 2018; Lee and Choi 2018; Yao et al. 2019b) or directly use a meta-optimizer to learn the optimization process (Ravi and Larochelle 2016); (2) metric-based few-shot learning…”
Contextualizing Multiple Tasks via Learning to Decompose (2021, methodology): “[38, 54, 68, 58, 59] adapts the feature embeddings or the model initialization given a few-shot support set, [48, 57] compose instance representations with the supervision of attributes, and [4] utilizes the ﬁne-grained visual concepts.”