Neural Processes

By Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo Jimenez Rezende, S. M. Ali Eslami, Yee Whye Teh. 2018

617 citations recorded by the source corpus.

A neural network (NN) is a parameterised function that can be tuned via gradient descent to approximate a labelled collection of data with high precision. A Gaussian process (GP), on the other hand, is a probabilistic model that defines a distribution over possible functions, and is updated in light of data via the rules of probabilistic inference. GPs are probabilistic, data-efficient and flexible, however they are also computationally intensive and thus limited in their applicability. We introduce a class of neural latent variable models which we call Neural Processes (NPs), combining the best of both worlds. Like GPs, NPs define distributions over functions, are capable of rapid adaptation to new observations, and can estimate the uncertainty in their predictions. Like NNs, NPs are computationally efficient during training and evaluation but also learn to adapt their priors to data. We demonstrate the performance of NPs on a range of learning tasks, including regression and optimisation, and compare and contrast with related models in the literature.

Subfields: Artificial Intelligence, Model Architectures, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Probabilistic & Bayesian Methods

Citations in the corpus

Meta-Learning Probabilistic Inference for Prediction (2018, methodology): “…et al., 2018) to cover a broader class of methods, including gradient based meta-learning (Finn et al., 2017; Ravi and Larochelle, 2017), metric based meta-learning (Snell et al., 2017), amortized MAP inference (Qiao et al., 2017) and conditional probability modelling (Garnelo et al., 2018a;b).”
Occupancy Networks: Learning 3D Reconstruction in Function Space (2018, methodology): “We optimize a lower bound [21, 39, 59] to the negative log-likelihood of the generative model p((oij)j=1:K |(pij)j=1:K):”
Meta-Amortized Variational Inference and Learning (2019, background): “Although the Neural Process (NP) (Garnelo et al. 2018; Kim et al. 2019) is similar to our work in that it derives predictions for new targets by conditioning the encoder network on a relevant context set, it models uncertainty over a distribution of functions.”
Texture Fields: Learning Texture Representations in Function Space (2019, methodology): “ Several works [6,21,24] proposed to formulate 3D reconstruction as learning a continuous occupancy function or a signed-distance ﬁeld in 3D space, parameterized via a neural network. Furthermore, in [8], a continuous function for color values was used for generating 2D images. In this work, we exte”
Empirical Bayes Transductive Meta-Learning with Synthetic Gradients (2020, methodology): “This amortization technique was first introduced in the case of variational autoencoders (Kingma & Welling, 2013; Rezende et al., 2014), and has been extended to Bayesian inference in the case of neural processes (Garnelo et al., 2018).”
Attentive Neural Processes (2019, methodology): “Neural Processes (NPs) (Garnelo et al., 2018a;b) offer an efﬁcient method to modelling a distribution over regression functions, with prediction complexity linear in the context set size.”
GP-VAE: Deep Probabilistic Time Series Imputation (2019, background): “There are also some VAE-based imputation methods that are designed for a setting where the data is complete at training time and the missingness only occurs at test time [15, 16, 20].”
Continual Learning via Online Leverage Score Sampling (2019, background): “Recently, a number of approaches have been proposed to adapt a DNN model to the continual learning setting, from an adaptive model architecture perspective such as adding columns or neurons for new tasks [30, 38, 32]; model parameter adjustment or regularization techniques like, imposing restriction”
Meta reinforcement learning as task inference (2019, background): “Several works have considered meta-learning in a hierarchical Bayesian setting where “learning” about a task distribution is realized as inferring latent causes that are shared across tasks (or data sets) in a hierarchical probabilistic model (e.g. Fei-Fei et al. (2003); Garnelo et al. (2018b;a)).”
What Can Transformers Learn In-Context? A Case Study of Simple Function Classes (2022, background): “Most relevant to our setting are approaches that directly take as input examples from a downstream task and a query input and produce the corresponding output [26, 42, 61, 24, 23, 32].”
Deep Amortized Clustering (2019, background): “In a sense this shares a similar philosophy as Neural Processes (Garnelo et al., 2018b;a), which meta-learns from multiple datasets to learn a prior over functions.”
Variational Few-Shot Learning (2019, methodology): “Unlike the strategy ( i.e ., generate distributions with an averaged feature of a mini-batch) frequently adopted in [21, 14, 8], which implies every sample weighs the same, our method uniﬁes distribution of every single data into the ﬁnal one.”
Transformers Can Do Bayesian Inference (2021, background): “More recently (Conditional) Neural Processes (Garnelo et al., 2018a;b; Kim et al., 2018) were proposed for meta-learning.”
Functional Variational Bayesian Neural Networks (2019, background): “Neural Processes (NP) (Garnelo et al., 2018) try to model any conditional distribution given arbitrary data points, whose prior is specified implicitly by prior samples.”
On Modulating the Gradient for Meta-learning (2020, methodology): “Error (MSE) on the image regression tasks compared to Conditional Neural Process (CNP) [12], CAVIA [45], and MAML [9].”
Principled Uncertainty Estimation for Deep Neural Networks (2018, positive): “More recently, Conditional Neural Processes (Garnelo et al. 2018a) and Neural Processes (Garnelo et al. 2018b) instead model the function generating stochastic process via a neural network embedding and variational inference.”