Towards a Neural Statistician

By Harrison Edwards, Amos J. Storkey. 2016

445 citations recorded by the source corpus.

An efficient learner is one who reuses what they already know to tackle a new problem. For a machine learner, this means understanding the similarities amongst datasets. In order to do this, one must take seriously the idea of working with datasets, rather than datapoints, as the key objects to model. Towards this goal, we demonstrate an extension of a variational autoencoder that can learn a method for computing representations, or statistics, of datasets in an unsupervised fashion. The network is trained to produce statistics that encapsulate a generative model for each dataset. Hence the network enables efficient learning from new datasets for both unsupervised and supervised tasks. We show that we are able to learn statistics that can be used for: clustering datasets, transferring generative models to new datasets, selecting representative samples of datasets and classifying previously unseen classes. We refer to our model as a neural statistician, and by this we mean a neural network that can learn to compute summary statistics of datasets without supervision.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Signal Processing

Citations in the corpus

Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks (2017, methodology): “5% neural statistician (Edwards & Storkey, 2017) 98 .”
Toward Understanding Catastrophic Forgetting in Continual Learning (2019, methodology): “Other related efforts that can be used as alternatives to Task2Vec include, e.g., [12, 46, 50].”
One-Shot Imitation Learning (2017, background): “…has been studied for image recognition (Vinyals et al., 2016; Koch, 2015; Santoro et al., 2016; Ravi & Larochelle, 2017), generative modeling (Edwards & Storkey, 2017; Rezende et al., 2016), learning “fast” reinforcement learning agents with recurrent policies (Duan et al., 2016; Wang et…”
Meta-Learning Probabilistic Inference for Prediction (2018, methodology): “The form of these distributions is identical to those used in amortized variational inference (Edwards and Storkey, 2017; Kingma and Welling, 2014).”
Prototypical Networks for Few-shot Learning (2017, background): “We compare against various baselines, including the Neural Statistician [7], Meta-Learner LSTM [24], MAML [9], and both the fine-tuned and non-fine-tuned versions of Matching Networks [32].”
How to train your MAML (2018, methodology): “The application of meta-learning in the few-shot learning setting has enabled the overwhelming majority of the current state of the art few-shot learning methods (Vinyals et al., 2016; Ravi & Larochelle, 2016; Edwards & Storkey, 2016; Finn et al., 2017; Li et al., 2017; Munkhdalai & Yu, 2017).”
Data Augmentation Generative Adversarial Networks (2017, methodology): “…use a hierarchical Boltzmann machine, through to modern deep learning architectures for one-shot conditional generation in (Rezende et al., 2016), hierarchical variational autoencoders in (Edwards & Storkey, 2017) and most recently a GAN based one-shot generative model (Mehrotra & Dukkipati, 2017).”
Task Agnostic Meta-Learning for Few-Shot Learning (2018, neutral): “…no conv 91.37 ± 0.97% 96.84 ± 0.36% TAML(GE(2)), no conv 91.3 ± 1.0% 96.76 ± 0.4% TAML(Atkinson), no conv 91.77 ± 0.97% 97.0 ± 0.4% TAML (Gini-Coeﬃcient), no conv 93.17 ± 1.0% - Siamese Nets [17] 97.3% 98.4% Matching Nets [18] 98.1% 98.9% Neural Statistician [20] 98.1% 99.5% Memory Mod.”
Relational inductive biases, deep learning, and graph networks (2018, methodology): “Such an approach is the essence of the Deep Sets and related models (Zaheer et al., 2017; Edwards and Storkey, 2016; Pevn`y and Somol, 2017), which we explore further in Section 4.2.3.”
Set Transformer (2018, background): “Recently, Edwards & Storkey (2017) and Zaheer et al. (2017) propose neural network architectures which meet both criteria, which we call set pooling methods.”
Conditional Neural Processes (2018, methodology): “A model that is conceptually very similar to CNPs (and in particular the latent variable version) is the “neural statis-tician” paper (Edwards & Storkey, 2016) and the related variational homoencoder (Hewitt et al., 2018).”
Recasting Gradient-Based Meta-Learning as Hierarchical Bayes (2018, methodology): “A variety of inference methods have been used in Bayesian models, including exact inference (Lake et al., 2011), sampling meth-ods (Salakhutdinov et al., 2012), and variational methods (Edwards & Storkey, 2017).”
Probabilistic Model-Agnostic Meta-Learning (2018, methodology): “Our work is most related to methods that combine deep networks and probabilistic methods for few-shot learning [6, 15, 23].”
Low-Shot Learning from Imaginary Data (2018, background): “It might then accumulate statistics over the training set using recurrent neural networks (RNNs) [35, 23], memory-augmented networks [27], or multilayer perceptrons (MLPs) [6], perform gradient descent steps to finetune the representation [9], and/or collapse each class into prototypes [30].”
Learning to Compare: Relation Network for Few-Shot Learning (2017, methodology): “This is despite that many alternatives have significantly more complicated machinery [27, 8], or fine-tune on the target problem [10, 39], while we do not.”
Few-Shot Learning with Graph Neural Networks (2017, neutral): “1-shot 5-shot 1-shot 5-shot Pixels Vinyals et al. (2016) 41.7% 63.2% 26.7% 42.6% Siamese Net Koch et al. (2015) 97.3% 98.4% 88.2% 97.0% Matching Networks Vinyals et al. (2016) 98.1% 98.9% 93.8% 98.5% N. Statistician Edwards & Storkey (2016) 98.1% 99.5% 93.2% 98.1% Res.”
Meta-learning autoencoders for few-shot prediction (2018, methodology): “The recent neural statistician work [5] applies the variational autoencoder approach to the encoding and generation of datasets.”
Uncertainty in Multitask Transfer Learning (2018, background): “More recently, Edwards and Storkey [10] and Bouchacourt et al. [6] explore hierarchical Bayesian inference with neural networks and evaluate on new tasks.”
TADAM: Task dependent adaptive metric for improved few-shot learning (2018, background): “The third approach relies on Bayesian modeling of the prior distribution of the different categories like in Li et al. [15], Bauer et al. [1], or Lake et al. [13], Edwards and Storkey [4], Lacoste et al. [12] who rely on hierarchical Bayesian modeling.”