Transformation Autoregressive Networks

By Junier B. Oliva, Kumar Avinava Dubey, Manzil Zaheer, Barnab'as P'oczos, Ruslan Salakhutdinov, Eric P. Xing, Jeff Schneider. 2018

88 citations recorded by the source corpus.

The fundamental task of general density estimation $p(x)$ has been of keen interest to machine learning. In this work, we attempt to systematically characterize methods for density estimation. Broadly speaking, most of the existing methods can be categorized into either using: \textit{a}) autoregressive models to estimate the conditional factors of the chain rule, $p(x_{i}\, |\, x_{i-1}, \ldots)$; or \textit{b}) non-linear transformations of variables of a simple base distribution. Based on the study of the characteristics of these categories, we propose multiple novel methods for each category. For example we proposed RNN based transformations to model non-Markovian dependencies. Further, through a comprehensive study over both real world and synthetic data, we show for that jointly leveraging transformations of variables and autoregressive conditional models, results in a considerable improvement in performance. We illustrate the use of our models in outlier detection and image modeling. Finally we introduce a novel data driven framework for learning a family of distributions.

Subfields: Artificial Intelligence, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Model Architectures

Citations in the corpus

FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models (2018, methodology): “By using an autoregressive model and specifying an ordering of the dimensions, the Jacobian of f is enforced to be lower triangular (Kingma et al., 2016; Oliva et al., 2018).”
High-Dimensional Multivariate Forecasting with Low-Rank Gaussian Copula Processes (2019, methodology): “The GARCH [21] (Generalize Orthogonal - Generalize Autoregressive Conditional Heteroskedastic-ity) model is a composit model providing dynamics for the conditional mean and conditional covariance matrix of a multivariate system.”
Neural Spline Flows (2019, background): “[41] generalized permutations to a more general class of linear transformations, and Kingma and Dhariwal [28] demonstrated improvements on a range of image tasks.”
Normalizing Flows for Probabilistic Modeling and Inference (2019, methodology): “Linear flows based on the PLU decomposition have been proposed by e.g. Oliva et al. (2018); Kingma and Dhariwal (2018).”
FloWaveNet : A Generative Flow for Raw Audio (2018, methodology): “, 2017) and transformation autoregressive network (Oliva et al., 2018).”
Neural Autoregressive Flows (2018, methodology): “…DDSF architecture in their best performing architecture for each task (keeping all other settings ﬁxed) results in substantial performance gains, and also outperforms the more recent Transformation Autoregressive Networks (TAN) Oliva et al. (2018), setting a new state-of-the-art for these tasks.”
Unconstrained Monotonic Neural Networks (2019, background): “From these results, we could conclude that Transformation Autoregressive Networks [TAN, Oliva et al., 2018] is overall the best method for density estimation.”
Deep Learning for Time Series Forecasting: Tutorial and Literature Survey (2020, methodology): “, [46, 101, 137]) studies invertible NNs that typically transform isotropic Gaussians to more complex data distributions.”
Block Neural Autoregressive Flow (2019, background): “Additionally, Oliva et al. (2018) investigated different possibilities for the conditioner of an autoregressive transformation (e.g., recurrent neural networks).”
Autoregressive Energy Machines (2019, methodology): “In particular, it improves substantially on the the MADE-MoG results reported by Papamakarios et al. (2017), and improves on the state-of-the-art results reported by NAF (Huang et al., 2018) and TAN (Oliva et al., 2018).”
Point Cloud GAN (2018, background): “However, compared to supervised learning, unsupervised generative models for 3D data are still under explored (Achlioptas et al., 2017; Oliva et al., 2018).”
Cubic-Spline Flows (2019, methodology): “S MAF [23] 0:30 0:01 10:08 0:02 17:39 0:02 11:68 0:44 156:36 0:28 NAFy [13] 0:62 0:01 11:96 0:33 15:09 0:40 8:86 0:15 157:73 0:04 BLOCK-NAF [3] 0:61 01 12 06 09 14 71 38 8 95 07 157 36 03 TAN VARIOUS [22] 0:60 0:01 12:06 0:02 13:78 0:02 11:01 0:48 159:80 0:07 3.1. Synthetic datasets We ﬁrst demonstr”
Flow Models for Arbitrary Conditional Likelihoods (2019, background): “Other Transformations and Compositions Other transformations like element-wise leaky-ReLU transformation (Oliva et al. 2018) and reverse transformation (Dinh, Krueger, and Bengio 2014) are readily applicable to transform the unobserved covariates x u since they do not rely on the conditioning…”