Attentive Neural Processes

By Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, S. M. Ali Eslami, Dan Rosenbaum, Oriol Vinyals, Yee Whye Teh. 2019

540 citations recorded by the source corpus.

Neural Processes (NPs) (Garnelo et al 2018a;b) approach regression by learning to map a context set of observed input-output pairs to a distribution over regression functions. Each function models the distribution of the output given an input, conditioned on the context. NPs have the benefit of fitting observed data efficiently with linear complexity in the number of context input-output pairs, and can learn a wide family of conditional distributions; they learn predictive distributions conditioned on context sets of arbitrary size. Nonetheless, we show that NPs suffer a fundamental drawback of underfitting, giving inaccurate predictions at the inputs of the observed data they condition on. We address this issue by incorporating attention into NPs, allowing each input location to attend to the relevant context points for the prediction. We show that this greatly improves the accuracy of predictions, results in noticeably faster training, and expands the range of functions that can be modelled.

Subfields: Artificial Intelligence, Model Architectures, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Probabilistic & Bayesian Methods

Citations in the corpus

Set Transformer (2018, methodology): “Kim et al. (2019) proposed attention-based conditional regression, where self-attention is applied to the query sets.”
Graph-Based Continual Learning (2020, background): “…such as multi-head attention have been applied successfully in many stationary settings (Vaswani et al., 2017; Xu et al., 2015; Zhang et al., 2018; Kim et al., 2019), we note that naive applications of such techniques in continual learning suffer from catastrophic forgetting due to representation…”
Adaptive Deep Kernel Learning (2019, methodology): “Furthermore, in this family of methods, Conditional Neural Processes (CNP) [31] and Attentive Neural Processes (ANP) [32] are even more relevant to our study as both methods learn conditional stochastic processes parameterized by conditions derived from training data points.”
Learning to Learn Kernels with Variational Random Features (2020, unknown): “· The conditional prior p ( ω t | x , S t ) is implemented with a prior network which takes an aggregated representation by using the cross attention (Kim et al., 2019) between x and S t .”
Empirical Bayes Transductive Meta-Learning with Synthetic Gradients (2020, background): “Explicit transduction is less explored in meta-learning, the exception is Liu et al. (2018), who adapted the idea of label propagation (Zhu et al.”
Task-Agnostic Online Reinforcement Learning with an Infinite Mixture of Gaussian Processes (2020, methodology): “(a) Single dynamics model: We investigate the task performance of using a GP, a DNN, and an Attentative Neural Process (ANP) [42, 43] as the dynamics model.”
Meta-Amortized Variational Inference and Learning (2019, background): “Although the Neural Process (NP) (Garnelo et al. 2018; Kim et al. 2019) is similar to our work in that it derives predictions for new targets by conditioning the encoder network on a relevant context set, it models uncertainty over a distribution of functions.”
Meta-Learning with Shared Amortized Variational Inference (2020, unknown): “This typically intractable marginal likelihood, required for training and prediction, can be approximated using (amortized) variational inference (Garnelo et al., 2018; Kim et al., 2019), Monte Carlo sampling (Gordon et al.”
What Can Transformers Learn In-Context? A Case Study of Simple Function Classes (2022, background): “Nguyen and Grover [47] introduce Transformer neural processes, building on prior work on neural processes [24, 23, 30], and show that they achieve state-of-the art performance on tasks such as image completion and contextual multi-armed bandits.”
On the Limitations of Representing Functions on Sets (2019, background): “Kim et al. (2019) add an attention mechanism to neural processes to improve empirical performance.”
Probabilistic Symmetries and Invariant Neural Networks (2020, result): “W could be interpreted as providing additional shared “context” for the input-output relationship, a construct that has been useful in recent work (Edwards and Storkey, 2017; Kim et al., 2019).”
Transformers Can Do Bayesian Inference (2021, background): “In our first set of experiments, we study the capability of PFNs to perform Bayesian inference for the tractable case of Gaussian Processes (GPs) with fixed hyperparameters (where we can compare to ground truth data; Section 5.1) and the intractable cases of GPs with unknown hyperparameters (Section”
MetaSDF: Meta-learning Signed Distance Functions (2020, methodology): “Another promising line of future investigation are recently proposed attention-based set encoders [33], which have been shown to perform better than set encoders that aggregate latent codes via pooling, though the memory complexity of the attention mechanism makes conditioning on large (5k points) c”
Contextual Document Embeddings (2024, unknown): “This shows connections with the a large body of deep learning research such as the non-parametric transformer (NPT) (Kossen et al., 2022) and the subfield of Neural Processes (Garnelo et al., 2018; Kim et al., 2019; Nguyen & Grover, 2023).”