Meta-Learning with Memory-Augmented Neural Networks

By Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, Timothy P. Lillicrap. 2016

2,170 citations recorded by the source corpus.

Subfields: Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

Citations in the corpus

The Natural Language Decathlon: Multitask Learning as Question Answering (2018, background): “, 2016), augment models with special memory mechanisms (Santoro et al., 2016; Schmidhuber, 1992), and maximize the degree to which models can learn new tasks (Finn et al.”
Few-shot Autoregressive Density Estimation: Towards Learning to Learn Distributions (2017, methodology): “This setting, called few-shot learning, has been studied using deep neural networks and many other approaches in the context of discriminative models, for example Vinyals et al. (2016); Santoro et al. (2016).”
Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks (2017, methodology): “The Omniglot dataset is augmented with rotations by multiples of 90 degrees, as proposed by Santoro et al. (2016).”
Matching Networks for One Shot Learning (2016, methodology): “Following [21], we augmented the data set with random rotations by multiples of 90 degrees and used 1200 characters for training, and the remaining character classes for evaluation.”
Shortcut learning in deep neural networks (2020, background): “y also should not be biased against minority groups. Meta-learning Meta-learning seeks to learn how to learn. An intermediate goal is to learn representations that can adapt quickly to new conditions [136,137,138]. This ability is connected to the identiﬁcation of causal graphs [139] since learning ”
Learning with Latent Language (2017, background): “ argmin (c) 2Rb X j L f(x(c) j ; ; (c));y(c) j (3) on the new dataset, then make predictions for new inputs using f(x(e); ;(c)). Closely related meta-learning approaches (e.g. Schmidhuber, 1987; Santoro et al., 2016; Vinyals et al., 2016) make use of the same data, but collapse the inner optimizatio”
One-Shot Imitation Learning (2017, background): “One-shot and few-shot learning has been studied for image recognition (Vinyals et al., 2016; Koch, 2015; Santoro et al., 2016; Ravi & Larochelle, 2017), generative modeling (Edwards & Storkey, 2017; Rezende et al., 2016), learning “fast” reinforcement learning agents with recurrent policies (Duan…”
Repurposing Pretrained Models for Robust Out-of-domain Few-Shot Learning (2021, neutral): “Second, memory-based approaches utilize previous tasks for new tasks with external memories (Santoro et al., 2016; Mishra et al., 2018).”
Meta-learning with differentiable closed-form solvers (2018, methodology): “, 2017) and learning how to access a memory module (Santoro et al., 2016; Kaiser et al., 2017; Munkhdalai & Yu, 2017), are fast but rely solely on the quality of the similarity metric, with no additional data-dependent adaptation at test-time.”
Prototypical Networks for Few-shot Learning (2017, methodology): “Since the meta-data vector and query point come from different input domains, we found it was helpful empirically to fix the prototype embedding g to have unit length, however we do not constrain the query embedding f .”
Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples (2019, methodology): “That said, there are other methods that less directly relate to the methods above, such as the work of Santoro et al. (2016) based on a memory-augmented recurrent network (one of the earliest meta-learning approach to few-shot learning).”
On First-Order Meta-Learning Algorithms (2018, background): “[16] on few-shot classification, and Duan et al.”
Backdoor Attacks on Federated Meta-Learning (2020, methodology): “Similarly to previous work [9, 8, 26], we resize images to 28× 28 and augment classes 4× through rotations by multiples of 90◦: we use 1200 classes for meta-training (split among the 4 users) and 418 for meta-testing; for each meta-training class, we hold out 5 examples for validation.”
Auto-Meta: Automated Gradient Based Meta Learner Search (2018, methodology): “In particular, RNN based methods, augmented with memory-augment network[22] or simple attention mechanism[16] have been applied to few-shot image classification tasks.”
Meta-SGD: Learning to Learn Quickly for Few Shot Learning (2017, methodology): “A recurrent model such as Long Short-Term Memory (LSTM) [11] processes data sequentially and figures out its own learning strategy from scratch in the course [20].”
Meta Networks (2017, methodology): “Santoro et al. (2016) proposed a memory-based approach and trained Neural Turing Machines (Graves et al., 2014) for one-shot learning, although the meta-learner and the one-shot learner in this work are not separable explicitly.”
Learning to reinforcement learn (2016, background): “For example, in a regression task, on each step the network receives as input an x value for which it is desired to output the corresponding y , but the network also receives an input disclosing the target y value for the preceding step (see Hochreiter et al., 2001; Santoro et al., 2016).”