Learning Deep Architectures for AI

By Yoshua Bengio. 2007

8,849 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

Citations in the corpus

Generative Adversarial Nets (2014, background): “The promise of deep learning is to discover rich, hierarchical models [2] that represent probability distributions over the kinds of data encountered in artificial intelligence applications, such as natural images, audio waveforms containing speech, and symbols in natural language corpora.”
GloVe: Global Vectors for Word Representation (2014, methodology): “This evaluation scheme favors models that produce dimensions of meaning, thereby capturing the multi-clustering idea of distributed representations (Bengio, 2009).”
FitNets: Hints for Thin Deep Nets (2014, methodology): “However, Bengio (2009) uses hand-deﬁned heuristics to measure the “simplicity” of an example in a sequence and Gulcehre & Bengio (2013)’s guidance hints require some prior knowledge of the end-task.”
Exact solutions to the nonlinear dynamics of learning in deep linear neural networks (2013, methodology): “We consider the effect of using autoencoders as the unsupervised pretraining module [18, 12], for which the input-output correlation matrix Σ(31) is simply the input correlation matrix Σ(11).”
InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets (2016, background): “A popular framework for unsupervised learning is that of representation learning [1, 2], whose goal is to use unlabelled data to learn a representation that exposes important semantic features as easily decodable factors.”
Intriguing properties of neural networks (2013, background): “It has been argued [2] that the deep stack of non-linear layers in between the input and the output unit of a neural network are a way for the model to encode a non-local generalization prior over the input space.”
Deep Generative Stochastic Networks Trainable by Backprop (2013, neutral): “Research in deep learning (see Bengio (2009) and Ben-gio et al. (2013a) for reviews) grew from breakthroughs in unsupervised learning of representations, based mostly on the Restricted Boltzmann Machine (RBM) (Hinton et al., 2006), auto-encoder variants (Bengio et al., 2007; Vin-cent et al., 2008),…”
Better Mixing via Deep Representations (2012, methodology): “There is another – less commonly discussed – motivation for deep representations, introduced in Bengio (2009): the idea that they may, to some extent, help to disentangle the underlying factors of variation.”
A Generative Process for Contractive Auto-Encoders (2012, background): “A more general approach that we propose to investigate is based on the idea of learning a representation that does not throw these variations away but instead disentangles them from each other (Bengio, 2009).”
Neural Autoregressive Distribution Estimation (2016, background): “Deep neural networks (Bengio, 2009; LeCun et al., 2015) are at the core of state-of-the-art models for supervised tasks like image recognition (Krizhevsky et al., 2012) and speech recognition (Dahl et al., 2013).”
Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning (2018, methodology): “It has accelerated with the success of Deep Neural Networks (Bengio, 2009) and the availability of annotated data as offered by datasets such as Flickr30K (Young et al., 2014) and MS-COCO (Lin et al., 2014).”
Learning Features by Watching Objects Move (2016, methodology): “ing the input [4, 20, 44], predicting the pixels of the next frame in a video stream [17], metric learning on object track endpoints [46], temporally ordering shuffled frames from a video [29], and spatially ordering patches from a static image [8, 30].”
Context Encoders: Feature Learning by Inpainting (2016, background): “The context encoder is closely related to autoencoders [3, 20], as it shares a similar encoder-decoder architecture.”
GSNs : Generative Stochastic Networks (2015, background): “Research in deep learning (see Bengio, 2009; Bengio et al., 2013a for reviews) grew from breakthroughs in unsupervised learning of representations, based mostly on the restricted Boltzmann machine (RBM) (Hinton et al., 2006), auto-encoder variants (Bengio et al., 2007; Vincent et al., 2008) and…”
NICE: Non-linear Independent Components Estimation (2014, background): “Deep learning approaches (Bengio, 2009) rely on the learning of a representation of the data that would capture its most important factors of variation.”
Generative adversarial networks (2014, neutral): “monstrate the potential of the framework through qualitative and quantitative evaluation of the generated samples. 1 Introduction The promise of deep learning is to discover rich, hierarchical models [2] that represent probability distributions over the kinds of data encountered in artiﬁcial intelli”
Value Iteration Networks (2016, background): “Convolutional neural networks (CNNs; LeCun et al. 1998; Bengio 2009) are neural networks with a particular architecture that has proved useful for computer vision, among other domains.”
Representation Learning: A Review and New Perspectives (2012, background): “It has been extensively reviewed and discussed in Bengio (2009).”
Action-Conditional Video Prediction using Deep Networks in Atari Games (2015, background): “1 Introduction Over the years, deep learning approaches (see [5, 26] for survey) have shown great success in many visual perception problems (e.”