Modeling High-Dimensional Discrete Data with Multi-Layer Neural Networks

By Yoshua Bengio, Samy Bengio. 1999

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

Citations in the corpus

XLNet: Generalized Autoregressive Pretraining for Language Understanding (2019, background): “With a deep root in density estimation 4 [4, 32, 24], language modeling has been a rapidly-developing research area [9, 1, 3].”
Neural Autoregressive Distribution Estimation (2016, methodology): “• MADE (Germain et al., 2015): Generalization of the neural network approach of Bengio and Bengio (2000), to multiple layers.”
MADE: Masked Autoencoder for Distribution Estimation (2015, methodology): “Previous work on autoregressive neural networks have also found it advantageous to use direct connections between the input and output layers (Bengio & Bengio, 2000).”
Image Transformer (2018, methodology): “This work is strongly inspired by autoregressive models such as fully visible belief networks and NADE (Bengio & Bengio, 2000; Larochelle & Mur-ray, 2011) in that we also factor the joint probability of the image pixels into conditional distributions.”
Density estimation using Real NVP (2016, background): “Autoregressive models [18, 6, 37, 20] can implement this strategy while typically retaining a great deal of flexibility.”
Towards Conceptual Compression (2016, methodology): “…to use latents and it-erate sequentially over the pixels, predicting a given pixel from the previous ones (from top left to bottom right) in an autoregressive fashion (Bengio & Bengio, 1999; Graves & Schmidhuber, 2009; Larochelle & Murray, 2011; Gre-gor & LeCun, 2011; van den Oord et al., 2016).”
NICE: Non-linear Independent Components Estimation (2014, methodology): “The triangular structure used in NICE to obtain tractability is also present in another tractable density model, the neural autoregressive density estimator (NADE) (Larochelle and Murray, 2011), inspired by (Bengio and Bengio, 2000).”
RNADE: The real-valued neural autoregressive density-estimator (2013, methodology): “Neural networks can also be used for each binary prediction task [8].”
A Deep and Tractable Density Estimator (2013, methodology): “For the binary data case, Frey (1998) considered the use of logistic regression conditional models, while Bengio & Bengio (2000) proposed a single layer neural network architecture, with a parameter sharing scheme different from the one in the NADE model (Larochelle & Murray, 2011).”
SampleRNN: An Unconditional End-to-End Neural Audio Generation Model (2016, methodology): “Our work is related to earlier work on auto-regressive multi-layer neural networks, starting with Bengio & Bengio (1999), then NADE (Larochelle & Murray, 2011) and more recently Pix-elRNN (van den Oord et al., 2016).”
Normalizing Flows for Probabilistic Modeling and Inference (2019, background): “Alongside normalizing flows, another popular class of models for high-dimensional distributions is the class of autoregressive models (Bengio and Bengio, 2000; Uria et al., 2016).”
Reweighted Wake-Sleep (2014, background): “Deep AutoRegressive Network (DARN): An autoregressive sigmoid belief network layer (Frey, 1998; Bengio and Bengio, 2000; Gregor et al. , 2014) is similar to an SBN layer but with the output units x i not being independent of each other, given the layer’s input y .”
Iterative Neural Autoregressive Distribution Estimator NADE-k (2014, methodology): “On the other hand, this paper follows up on the Neural Autoregressive Distribution Estimator (NADE, Larochelle and Murray, 2011), which specializes previous neural auto-regressive density estimators (Bengio and Bengio, 2000) and was recently extended (Uria et al. , 2014) to deeper architectures.”
Vision-Based Multi-Task Manipulation for Inexpensive Robots Using End-to-End Learning from Demonstration (2017, methodology): “This idea of modeling the joint probability distribution of outputs by casting it as a product of conditional distributions is used in Pixel RNNs [43], Neural Autoregressive Distribution Estimator (NADE) [35], and fully visible neural networks [44], [45].”
Deep Computation Model for Unsupervised Feature Learning on Big Data (2016, background): “Index Terms—Big data, deep learning model, tensor auto-encoder, back-propagation algorithm, feature learning Ç”
A Neural Autoregressive Topic Model (2012, background): “In fact, other choices not inspired by the RBM have proven less successful (see Bengio and Bengio [4] and Larochelle and Murray [3] for a discussion).”
Effect of Non-linear Deep Architecture in Sequence Labeling (2013, background): “Although some studies argue that non-linear neural nets suffer less from the curse of dimensionality (Attali and Pag´es, 1997; Bengio and Bengio, 2000; Pitkow, 2012), counter arguments have been offered (Camastra, 2003; Verleysen et al., 2003).”