Conditional Neural Processes

By Marta Garnelo, Dan Rosenbaum, Chris J. Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Jimenez Rezende, S. M. Ali Eslami. 2018

864 citations recorded by the source corpus.

Deep neural networks excel at function approximation, yet they are typically trained from scratch for each new function. On the other hand, Bayesian methods, such as Gaussian Processes (GPs), exploit prior knowledge to quickly infer the shape of a new function at test time. Yet GPs are computationally expensive, and it can be hard to design appropriate priors. In this paper we propose a family of neural models, Conditional Neural Processes (CNPs), that combine the benefits of both. CNPs are inspired by the flexibility of stochastic processes such as GPs, but are structured as neural networks and trained via gradient descent. CNPs make accurate predictions after observing only a handful of training data points, yet scale to complex functions and large datasets. We demonstrate the performance and versatility of the approach on a range of canonical machine learning tasks, including regression, classification and image completion.

Subfields: Artificial Intelligence, Model Architectures, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Probabilistic & Bayesian Methods

Citations in the corpus

Fast and Flexible Multi-Task Classification Using Conditional Neural Adaptive Processes (2019, methodology): “Whereas previous work on CNPs has focused on homogeneous regression and classification datasets and fairly simple models, here we study multiple heterogeneous classification datasets and use a more complex model to handle this scenario.”
Meta-Learning Probabilistic Inference for Prediction (2018, methodology): “…et al., 2018) to cover a broader class of methods, including gradient based meta-learning (Finn et al., 2017; Ravi and Larochelle, 2017), metric based meta-learning (Snell et al., 2017), amortized MAP inference (Qiao et al., 2017) and conditional probability modelling (Garnelo et al., 2018a;b).”
Time-series forecasting with deep learning: a survey (2020, background): “In addition, Gaussian processes [21] have been extensively used for time-series prediction—with recent extensions including deep Gaussian processes [22], and parallels in deep learning via neural processes [23].”
Meta-Learning with Latent Embedding Optimization (2018, background): “The latter can be classified as metric-based [18, 40, 37], memory-based [35, 32], model-based [27, 26, 8, 9], and optimization-based [6, 7, 29].”
Set Transformer (2018, background): “, 2017), and conditional regression and classification (Garnelo et al., 2018).”
Embracing Change: Continual Learning in Deep Neural Networks. (2020, background): “In [86,87] both loops use gradient descent , while inner-loops can also be nearest-neighbor matching [88,89], recurrent neural representations [8,90,91], probabilistic inference [82,92], or mixed approaches [93 – 96].”
Neural Processes (2018, methodology): “NPs can therefore be described as meta-learning algorithms for few-shot function regression, although as shown in Garnelo et al. (2018) they can also be applied to few-shot learning tasks beyond regression.”
Task Agnostic Continual Learning via Meta Learning (2019, background): “For example, Conditional Neural Processes (CNP) (Garnelo et al., 2018) embed the observation and target pairs in context data ( x i , y i ) ∈ D ctx t by an encoder network r i = h ( x i , y i ; θ h ) .”
Meta-Learning in Neural Networks: A Survey (2020, background): “Neural Processes [166] aim to combine the uncertainty quantiﬁcation of Gaussian Processes with the versatility of neural networks, but they are not shown to work on modern few-shot benchmarks.”
Editable Neural Networks (2020, methodology): “For each editor We also compare to alternative editor function inspired by Conditional Neural Processes (CNP) (Garnelo et al. (2018)) that we refer to as Editable+CNP .”
Fast Context Adaptation via Meta-Learning (2018, methodology): “CAVIA is conceptually related to embedding-based approaches for fast adaptation such as conditional neural processes (CNPs) (Garnelo et al., 2018) and meta-learning with latent embedding optimisation (LEO) (Rusu et al.”
MetaSDF: Meta-learning Signed Distance Functions (2020, methodology): “Second, we compare to a conditional neural process (CNP) [6], representative of permutation-invariant set-encoders.”
Improved Few-Shot Visual Classification (2019, positive): “CNAPS is a few-shot adaptive classiﬁer based on conditional neural processes (CNP) [6].”
A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges (2020, methodology): “Inspired by the Gaussian process [133], πVAE is a stochastic process that learns the distribution over functions.”
Adaptive Risk Minimization: A Meta-Learning Approach for Tackling Group Shift (2020, methodology): “We instantiate the model and adaptation procedure based on a contextual meta-learning approach with deep neural networks [69, 20].”
Deep Amortized Clustering (2019, methodology): “In a sense this shares a similar philosophy as Neural Processes (Garnelo et al., 2018b;a), which meta-learns from multiple datasets to learn a prior over functions.”
What Can Transformers Learn In-Context? A Case Study of Simple Function Classes (2022, background): “Most relevant to our setting are approaches that directly take as input examples from a downstream task and a query input and produce the corresponding output [26, 42, 61, 24, 23, 32].”
Implicit Neural Representations with Periodic Activation Functions (2020, methodology): “Table 1: Quantitative comparison to Conditional Neural Processes [57] (CNPs) on the 32 × 32 CelebA test set.”