One-shot Learning with Memory-Augmented Neural Networks

By Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, Timothy P. Lillicrap. 2016

540 citations recorded by the source corpus.

Despite recent breakthroughs in the applications of deep neural networks, one setting that presents a persistent challenge is that of “one-shot learning.” Traditional gradient-based networks require a lot of data to learn, often through extensive iterative training. When new data is encountered, the models must inefficiently relearn their parameters to adequately incorporate the new information without catastrophic interference. Architectures with augmented memory capacities, such as Neural Turing Machines (NTMs), offer the ability to quickly encode and retrieve new information, and hence can potentially obviate the downsides of conventional models. Here, we demonstrate the ability of a memory-augmented neural network to rapidly assimilate new data, and leverage this data to make accurate predictions after only a few samples. We also introduce a new method for accessing an external memory that focuses on memory content, unlike previous methods that additionally use memory locationbased focusing mechanisms.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Ocean Engineering, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Ocean Engineering

In Representation & Self-Supervised Learning

Citations in the corpus

RL$^2$: Fast Reinforcement Learning via Slow Reinforcement Learning (2016, background): “Our work draws inspiration from a particular line of work (Younger et al., 2001; Santoro et al., 2016; Vinyals et al., 2016), which formulates meta-learning as an optimization problem, and can thus be optimized end-to-end via gradient descent.”
Growing a Brain: Fine-Tuning by Increasing Model Capacity (2017, background): “Transfer learning is related to both multi-task learning [32, 4, 28, 10, 11, 45, 24, 2] and learning novel categories from few examples [49, 19, 21, 35, 51, 22, 5, 13, 50, 47, 31].”
Reformer: The Efficient Transformer (2020, background): “The requirement that the memory be ﬁxed before has been removed in Santoro et al. (2016) at the cost of memory size and later alleviated by Rae et al. (2016).”
What can the brain teach us about building artificial intelligence? (2016, background): “Indeed, recent work on meta-learning (or learning-to-learn) in deep learning models provides a base for making good on this claim (Santoro et al, 2016; Vinyals et al, 2016; Bartunov and Vetrov, 2016).”
Neural Discrete Representation Learning (2017, background): “At the same time, challenging tasks such as few-shot learning [34], domain adaptation [17], or reinforcement learning [35] heavily rely on learnt representations from raw data, but the usefulness of generic representations trained in an unsupervised fashion is still far from being the dominant…”
Comparison of Maximum Likelihood and GAN-based training of Real NVPs (2017, methodology): “We will take ideas from supervised setting where people used a short-term memory to provide one-shot learning (Santoro et al., 2016; Vinyals et al., 2016).”
Hierarchical Multiscale Recurrent Neural Networks (2016, background): “A stack of representation layers, learned from the data in a way to optimize a supervised or unsupervised target task, make deep neural networks entertain advantages such as generalization to unseen examples (Hoffman et al., 2013; Santoro et al., 2016), sharing learned knowledge among multiple tasks”
Zero-shot Text Classification With Generative Language Models (2019, methodology): “However, parallel work such as Memory Augmented Neural Networks (Santoro et al., 2016) and Simple Neural Attentive Learners (Mishra et al., 2017) demonstrate the effectiveness of architecture based meta-learning.”
Meta-Learning for Semi-Supervised Few-Shot Classification (2018, background): “, 2017) or a generic inference network (Santoro et al., 2016; Mishra et al., 2017).”
Data Augmentation Generative Adversarial Networks (2017, methodology): “Various forms of memory augmented networks have also been used to collate the critical sparse information in an incremental way (Santoro et al., 2016; Vinyals et al., 2016).”
Task Agnostic Meta-Learning for Few-Shot Learning (2018, methodology): “Methods 5-way 1-shot 5-shot MANN, no conv [10] 82.”
Towards a Neural Statistician (2016, methodology): “We find that the performance is remarkably good, especially considering that the model itself was not discriminatively trained.”
Conditional Neural Processes (2018, methodology): “Results on the same task for MANN (Santoro et al., 2016), and matching networks (MN) (Vinyals et al., 2016) and CNP. using a signiﬁcantly simpler architecture (three convolutional layers for the encoder and a three-layer MLP for the decoder) and with a lower runtime of O ( n + m ) at test time as…”
Dynamic Few-Shot Visual Learning Without Forgetting (2018, background): “Meta-learning approaches typical involve a meta-learner model that given a few training examples of a new task it tries to quickly learn a learner model that “solves” this new task [20, 24, 1, 13, 19].”
Learning Algorithms for Active Learning (2017, methodology): “Our model builds on the matching-networks (MN) architecture presented by Vinyals et al. (2016), which enables “one-shot” learning, i.e. learning the appearance of a class from just a single example of that class (Santoro et al., 2016; Koch, 2015).”
Gradient Episodic Memory for Continual Learning (2017, background): “Zero-shot learning [Lampert et al., 2009, Palatucci et al., 2009] and one-shot learning [Fei-Fei et al., 2003, Vinyals et al., 2016, Santoro et al., 2016, Bertinetto et al., 2016] aim at performing well on unseen tasks, but ignore the catastrophic forgetting of previously learned tasks.”
Incremental Few-Shot Object Detection (2020, neutral): “A large body of FSL work has investigated how to learn from such scarce data without overfitting [34, 44, 6, 26, 7, 17, 47, 55, 50, 27, 39, 11, 16].”
Neural Processes (2018, neutral): “…example is matching networks (Vinyals et al., 2016), but there is a large literature of similar models for classiﬁcation (Koch et al., 2015; Santoro et al., 2016), reinforcement learning (Wang et al., 2016), parameter update (Finn et al., 2017; 2018), natural language processing (Bowman et…”
Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilization (2018, background): “This process of using past knowledge to rapidly solve new tasks is referred to as transfer learning, and several groups have recently proposed how ANNs can implement this form of rapid learning [22, 23, 24].”
Incremental Few-Shot Learning with Attention Attractor Networks (2018, background): “Santoro et al. (2016) use a read/write augmented memory, and Mishra et al. (2018) combine soft attention with temporal convolutions which enables retrieval of information from past episodes.”