Scaling learning algorithms towards AI

By Yoshua Bengio, Yann LeCun. 2007

1,387 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Scaling & Learning Theory

Citations in the corpus

Exact solutions to the nonlinear dynamics of learning in deep linear neural networks (2013, background): “These successes have been achieved despite the noted difficulty of training such deep architectures [7, 8, 9, 10, 11].”
Better Mixing via Deep Representations (2012, background): “specific function families have shown that choosing a sufficient depth of representation can yield exponential benefits, in terms of size of the model, to represent some functions (H̊astad, 1986; H̊astad and Goldmann, 1991; Bengio et al., 2006; Bengio and LeCun, 2007; Bengio and Delalleau, 2011).”
Efficient Estimation of Word Representations in Vector Space (2013, background): “Recurrent neural network based language model has been proposed to overcome certain limitations of the feedforward NNLM, such as the need to specify the context length (the order of the model N ), and because theoretically RNNs can efficiently represent more complex patterns than the shallow neural ”
Wide Residual Networks (2016, neutral): “Our code is available at https://github.com/szagoruyko/wide-residual-networks.”
Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets (2015, background): “…deep networks: their similarity to the hierarchical, recurrent structure of the neo-cortex and the theoretical justiﬁcation that certain patterns are more efﬁciently represented by functions employing multiple non-linearities instead of a single one (Minsky & Seymour, 1969; Bengio & LeCun, 2007).”
Representation Learning: A Review and New Perspectives (2012, background): “In Bengio and LeCun (2007), one of us introduced the notion of AI-tasks, which are challenging for current machine learning algorithms, and involve complex but highly structured dependencies.”
Building high-level features using large scale unsupervised learning (2011, methodology): “We also compare our method against several other algorithms such as deep autoencoders (Hinton & Salakhutdinov, 2006; Bengio et al., 2007) and K-means (Coates et al., 2011).”
Adversarial Examples: Attacks and Defenses for Deep Learning (2017, background): “For conventional ML algorithms, it is difficult to extract well-represented features due to limitations, such as curse of dimensionality [37], computational bottleneck [38], and requirement of the domain and expert knowledge.”
On the Transfer of Inductive Bias from Simulation to the Real World: a New Disentanglement Dataset (2019, background): “Learning well-disentangled representations of complex sensory data has been identified as one of the key challenges in the quest for AI [1, 40, 26, 2, 42, 24, 47], since they should contain all the information present in the observations in a compact and interpretable structure [1, 21, 6] while bein”
On the Fairness of Disentangled Representations (2019, background): “Indeed, isolating each independent factor of variation into the independent components of a representation vector should make it both interpretable and simplify downstream prediction tasks [6, 7, 29, 35, 56, 58, 60, 74, 82, 87, 89, 90, 1, 28].”
Are Disentangled Representations Helpful for Abstract Visual Reasoning? (2019, background): “Learning good representations of high-dimensional sensory data is of fundamental importance to Artificial Intelligence [4, 3, 6, 49, 7, 69, 67, 50, 59, 73].”
Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations (2018, background): “It has been argued that an artificial intelligence able to understand and reason about the world should be able to identify and disentangle the explanatory factors of variations hidden in the data [4, 41, 33, 3, 46, 31].”
Practical Recommendations for Gradient-Based Training of Deep Architectures (2012, background): “Theoretical results (H̊astad, 1986; H̊astad and Goldmann, 1991; Bengio et al., 2006b; Bengio and LeCun, 2007; Bengio and Delalleau, 2011) clearly identify families of functions where a deep representation can be exponentially more efficient than one that is insufficiently deep.”
An Empirical Study of Example Forgetting during Deep Neural Network Learning (2018, background): “Techniques of note — among others — are predeﬁned curricula of examples (Bengio & LeCun, 2007), self-paced learning (Kumar et al., 2010), and more recently meta-learning (Fan et al., 2017).”