A Simple Neural Attentive Meta-Learner

By Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, Pieter Abbeel. 2017

1,400 citations recorded by the source corpus.

Subfields: Representation & Self-Supervised Learning, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

In Optimization & Training

Citations in the corpus

Do Better ImageNet Models Transfer Better? (2018, methodology): “This problem is closely related to few-shot learning [39, 71, 55, 62, 21, 62, 49], but these methods are typically evaluated with training and test classes from the same distribution.”
Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need? (2020, methodology): “Following previous works [29, 34, 26, 41, 9], we use a ResNet12 as our backbone: the network consists of 4 residual blocks, where each has 3 convolutional layers with 3 × 3 kernel; a 2 × 2 max-pooling layer is applied after each of the ﬁrst 3 blocks; and a global average-pooling layer is on top of…”
Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design (2019, methodology): “Our best results were obtained by stacking convolutions and multi-head self attention into a gated residual network (Mishra et al., 2018; Chen et al., 2017), in a manner resembling the Transformer (Vaswani et al.”
PixelSNAIL: An Improved Autoregressive Generative Model (2017, methodology): “It projects the input to a lower dimensionality to produce the keys and values and then uses softmax-attention like in (Vaswani et al., 2017; Mishra et al., 2017) (masked so that the current pixel can only attend over previously generated pixels).”
Repurposing Pretrained Models for Robust Out-of-domain Few-Shot Learning (2021, methodology): “Second, memory-based approaches utilize previous tasks for new tasks with external memories (Santoro et al., 2016; Mishra et al., 2018).”
Meta-learning with differentiable closed-form solvers (2018, methodology): “For this reason, meta-learning methods applied to few-shot learning (e.g.ours, (Vinyals et al., 2016; Finn et al., 2017; Ravi & Larochelle, 2017; Mishra et al., 2018)) crucially take into account adaptation already during the training process to mimic the test-time setting, de facto learning how to…”
Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples (2019, methodology): “These were based on graph neural networks (Satorras & Estrach, 2018), relational networks (Sung et al., 2018) and attentional networks (Mishra et al., 2018).”
Meta-Learning with Latent Embedding Optimization (2018, background): “The latter can be classified as metric-based [18, 40, 37], memory-based [35, 32], model-based [27, 26, 8, 9], and optimization-based [6, 7, 29].”
Meta-Learning With Differentiable Convex Optimization (2019, methodology): “While gradient-based methods are general, they are prone to overfitting as the embedding dimension grows [18, 25].”
Automated Machine Learning: State-of-The-Art and Open Challenges (2019, background): “[87] proposed a meta-learner that tries to learn a common feature vector among different tasks.”
Auto-Meta: Automated Gradient Based Meta Learner Search (2018, methodology): “In particular, RNN based methods, augmented with memory-augment network[22] or simple attention mechanism[16] have been applied to few-shot image classification tasks.”
Transfer Learning with Neural AutoML (2018, background): “Recent meta-learning research has broadened this concept to learn generalizable representations across classes of tasks [27, 28].”
Task Agnostic Meta-Learning for Few-Shot Learning (2018, background): “[9] proposes a temporal convolution and attention based meta-learner called SNAIL that achieves state-of-the-art performance for few-shot classification tasks and reinforcement learning tasks.”
Sample-Efficient Reinforcement Learning via Counterfactual-Based Data Augmentation (2020, methodology): “For instance, context-based meta-RL methods adapt to new tasks by aggregating experience into a latent representation on which the policy is conditioned [8, 9, 10, 11].”
Set Transformer (2018, background): “Although not permutation invariant, (Mishra et al., 2018) has attention as one of its core components to meta-learn to solve various tasks using sequences of inputs.”
Relational Deep Reinforcement Learning (2018, background): “This approach builds off previous work suggesting that relational computations needn’t necessarily be biased by entities’ spatial proximity [20, 10, 21, 11, 13], and may also profit from iterative structured reasoning [22, 23, 24, 25].”