On the challenges of learning with inference networks on sparse, high-dimensional data

By Rahul G. Krishnan, Dawen Liang, Matthew D. Hoffman. 2017

90 citations recorded by the source corpus.

We study parameter estimation in Nonlinear Factor Analysis (NFA) where the generative model is parameterized by a deep neural network. Recent work has focused on learning such models using inference (or recognition) networks; we identify a crucial problem when modeling large, sparse, high-dimensional datasets -- underfitting. We study the extent of underfitting, highlighting that its severity increases with the sparsity of the data. We propose methods to tackle it via iterative optimization inspired by stochastic variational inference \citep{hoffman2013stochastic} and improvements in the sparse data representation used for inference. The proposed techniques drastically improve the ability of these powerful models to fit sparse data, achieving state-of-the-art results on a benchmark text-count dataset and excellent results on the task of top-N recommendation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Inference Suboptimality in Variational Autoencoders (2018, background): “Most relevant to our work is the recent work of Krishnan et al. (2017).”
Lagging Inference Networks and Posterior Collapse in Variational Autoencoders (2019, methodology): “Some approaches try to combine instance-speciﬁc variational inference with amortized variational inference to narrow this gap (Hjelm et al., 2016; Krishnan et al., 2018; Kim et al., 2018; Marino et al., 2018).”
Improving Inference for Neural Image Compression (2020, methodology): “The idea of combining amortized inference with iterative optimization has been studied and refined by various authors [Hjelm et al., 2016, Kim et al., 2018, Krishnan et al., 2018, Marino et al., 2018].”
Semi-Amortized Variational Autoencoders (2018, methodology): “Our work is most closely related the line of work which uses a separate model to initialize variational parameters and subsequently updates them through an iterative procedure (Salakhutdinov & Larochelle, 2010; Cho et al., 2013; Salimans et al., 2015; Hjelm et al., 2016; Krishnan et al., 2018; Pu et”
Latent Alignment and Variational Attention (2018, methodology): “A hybrid method such as semi-amortized inference [39, 34] might be a potential future direction worth exploring.”
Likelihood-free MCMC with Approximate Likelihood Ratios (2019, background): “However, this typically leads to an amortization gap [42, 43] which has been extensively studied by [44].”
COIN: COmpression with Implicit Neural representations (2021, background): “In a similar vein to work in the latent variable model literature [14, 19, 16, 22], several works [5, 12, 39] attempt to close the amortization gap [8] by performing iterative gradient-based optimization steps on top of the use of amortized inference networks.”