Deep Learning of Representations: Looking Forward

By Yoshua Bengio. 2013

715 citations recorded by the source corpus.

Deep learning research aims at discovering learning algorithms that discover multiple levels of distributed representations, with higher levels representing more abstract concepts. Although the study of deep learning has already led to impressive theoretical results, learning algorithms and breakthrough experiments, several challenges lie ahead. This paper proposes to examine some of these challenges, centering on the questions of scaling deep learning algorithms to much larger models and datasets, reducing optimization difficulties due to ill-conditioning or local minima, designing more efficient and powerful inference and sampling procedures, and learning to disentangle the factors of variation underlying the observed data. It also proposes a few forward-looking research directions aimed at overcoming these challenges.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation (2013, background): “Stochastic neurons with binary outputs are also interesting because they can easily give rise to sparse representations (that have many zeros), a form of regularization that has been used in many representation learning algorithms (Bengio et al. , 2013).”
Predicting Parameters in Deep Learning (2013, background): “A major drawback of this technique is that training is very inefficient in how it makes use of parallel resources [1].”
Learning Factored Representations in a Deep Mixture of Experts (2013, background): “By conditioning our gating and expert networks on the output of the previous layer, our model can express an exponentially large number of effective experts.”
Recent advances in convolutional neural networks (2015, methodology): “Parallelized SGD methods [20], [65], [66] improve SGD to”
MaskConnect: Connectivity Learning by Gradient Descent (2018, neutral): “Finally, our approach is also related to conditional computation methods [34,35,36,37,38,39,40,41,42,43], which learn to drop out blocks of units.”
Deep Learning: Methods and Applications (2014, methodology): “More recent work on autoencoders [22, 30] shows the capability of generalized denoising autoencoders in carrying out sampling, thus they may overcome the earlier problem of filling-in the missing modality’s information.”
Learning to Continually Learn (2020, methodology): “Gating functions for such conditional computation [3] have been learned via the REINFORCE algorithm [2] and at scale via a standard back-propagation in a sparsely-gated mixture-of-experts [38], but these works focus on computational capacity and efﬁciency rather than a r X i v : 2002 .”
A Hierarchical Recurrent Encoder-Decoder for Generative Context-Aware Query Suggestion (2015, methodology): “To obtain an order-sensitive representation of a query, we use a particular neural network architecture called Recurrent Neural Network (RNN) [3, 25].”
Self-Normalizing Neural Networks (2017, background): “Surprisingly, it is hard to ﬁnd success stories with FNNs that have many hidden layers, though they would allow for different levels of abstract representations of the input [2].”
Estimating or Propagating Gradients Through Stochastic Neurons (2013, background): “The symmetry-breaking and induced sparsity may also compensate for the extra variance and possibly help to reduce ill-conditioning, as hypothesized by Bengio (2013).”
Learning model-based planning from scratch (2017, methodology): “More recently, neural networks have been trained to perform “conditional” and “adaptive computation” [3, 2, 8], which results in a dynamic computational graph.”
Low-Rank Approximations for Conditional Feedforward Computation in Deep Neural Networks (2013, background): “This paper expands on the idea of conditional computation introd uce in [2], where the nodes of a deep network are augmented by a set of gating uni ts that determine when a node should be calculated.”
Demystifying Parallel and Distributed Deep Learning (2018, background): “Bengio [17] discusses scaling deep learning from various perspectives, focusing on models, optimization algorithms, and datasets.”
Metacontrol for Adaptive Imagination-Based Optimization (2017, background): “Our work is also related to recent notions of “conditional computation” (Bengio, 2013; Bengio et al., 2015), which adaptively modiﬁes network structure online, and “adaptive computation time” (Graves, 2016) which allows for variable numbers of internal “pondering” iterations to optimize…”
BlockDrop: Dynamic Inference Paths in Residual Networks (2017, methodology): “Several conditional computation methods have been proposed to dynamically execute different modules of a network model on a per-example basis [3, 2].”
Domain Adaptive Neural Networks for Object Recognition (2014, background): “A better model on raw pixels might be achieved by using deeper neural network layers with a similar strategy since deep architectures have brought some successes in many applications in recent years (Bengio, 2013).”