Iterative Amortized Inference

By Joseph Marino, Yisong Yue, Stephan Mandt. 2018

189 citations recorded by the source corpus.

Inference models are a key component in scaling variational inference to deep latent variable models, most notably as encoder networks in variational auto-encoders (VAEs). By replacing conventional optimization-based inference with a learned model, inference is amortized over data examples and therefore more computationally efficient. However, standard inference models are restricted to direct mappings from data to approximate posterior estimates. The failure of these models to reach fully optimized approximate posterior estimates results in an amortization gap. We aim toward closing this gap by proposing iterative inference models, which learn to perform inference optimization through repeatedly encoding gradients. Our approach generalizes standard inference models in VAEs and provides insight into several empirical findings, including top-down inference techniques. We demonstrate the inference optimization capabilities of iterative inference models and show that they outperform standard inference models on several benchmark data sets of images and text.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Probabilistic & Bayesian Methods

Citations in the corpus

Object-Centric Learning with Slot Attention (2020, methodology): “[45] Joseph Marino, Yisong Yue, and Stephan Mandt.”
NVAE: A Deep Hierarchical Variational Autoencoder (2020, background): “…VAEs [1, 2] is dedicated to the statistical challenges, such as reducing the gap between approximate and true posterior distributions [3, 4, 5, 6, 7, 8, 9, 10], formulating tighter bounds [11, 12, 13, 14], reducing the gradient noise [15, 16], extending VAEs to discrete variables [17, 18, 19,…”
Towards causal generative scene models via competition of experts (2020, methodology): “The recent IODINE model of Greff et al. (2019) instead uses a reﬁnement network (Marino et al., 2018) to perform iterative amortised variational inference over independent scene components which are separately decoded and then combined via a softmax to form the scene.”
Multi-Object Representation Learning with Iterative Variational Inference (2019, methodology): “Instead of amortizing the posterior directly (as in a regular VAE encoder), the reﬁnement network can be thought of as amortizing the gradient of the posterior (Marino et al., 2018a).”
Lagging Inference Networks and Posterior Collapse in Variational Autoencoders (2019, methodology): “Some approaches try to combine instance-speciﬁc variational inference with amortized variational inference to narrow this gap (Hjelm et al., 2016; Krishnan et al., 2018; Kim et al., 2018; Marino et al., 2018).”
COIN: COmpression with Implicit Neural representations (2021, background): “In a similar vein to work in the latent variable model literature [14, 19, 16, 22], several works [5, 12, 39] attempt to close the amortization gap [8] by performing iterative gradient-based optimization steps on top of the use of amortized inference networks.”
A General Method for Amortizing Variational Filtering (2018, methodology): “Following [32], we normalize the inputs to the inference model using layer normalization [2].”
VoxelMorph: A Learning Framework for Deformable Medical Image Registration (2018, background): “This process is sometimes referred to as amortized optimization [66].”
Semi-Amortized Variational Autoencoders (2018, methodology): “Marino et al. (2018) utilize meta-learning to train an inference network which learns to perform iterative inference by training a deep model to output the variational parameters for each time step.”
Advances in Variational Inference (2017, background): “Amortization can be also made iterative by feeding back the amortization error into the inference model [116].”
Better Set Representations For Relational Reasoning (2020, background): “Iterative inference is the general idea of optimizing a learning procedure to recover latent variables instead of directly mapping from the input to the latent variables in one step [12].”
Meta-Learning with Shared Amortized Variational Inference (2020, unknown): “Semi-amortized inference techniques (Marino et al., 2018; Kim et al., 2018), which combine feed-forward parameter initialization and iterative gradient-based reﬁnement of the approximate posterior, can be seen as a hybrid of optimization-based and Bayesian approaches.”
Likelihood-free MCMC with Approximate Likelihood Ratios (2019, background): “However, this typically leads to an amortization gap [42, 43] which has been extensively studied by [44].”
Learning Object-Centric Representations of Multi-Object Scenes from Multiple Views (2021, methodology): “Using a spatial mixture model [10] and iterative amortized inference [19], MulMON sidesteps the main technical difﬁculty of the multi-object-multi-view scenario—maintaining object correspondence across views— by iteratively updating the latent object representations for a scene over multiple views,…”
Meta-Learning an Inference Algorithm for Probabilistic Programs (2021, background): “Amortised inference and inference compilation (Gershman & Goodman, 2014; Le et al., 2017; Paige & Wood, 2016; Stuhlmüller et al., 2013; Kingma & Welling, 2013; Mnih & Gregor, 2014; Rezende et al., 2014; Ritchie et al., 2016; Marino et al., 2018) are closely related to our approach in that they also ”
Entity Abstraction in Visual Model-Based Reinforcement Learning (2019, methodology): “…binding problem into an inference problem with which we can build upon state-of-the-art techniques in amortized iterative variational inference [6–8] to use temporal continuity and interactive feedback to infer the posterior distribution of the entity variables given a sequence of observations…”
NeRF: Neural Radiance Field in 3D Vision, A Comprehensive Review (2022, unknown): “The NeRF renderer in NeRF-VAE was conditioned on latent code which was trained using Iterative Amortized Inference [113][114] and a ResNet [100] based encoder.”
Improving Inference for Neural Image Compression (2020, methodology): “The idea of combining amortized inference with iterative optimization has been studied and refined by various authors [Hjelm et al., 2016, Kim et al., 2018, Krishnan et al., 2018, Marino et al., 2018].”