Generalization and network design strategies

By Yann LeCun. 1989

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistical Learning Theory

Citations in the corpus

Better Mixing via Deep Representations (2012, methodology): “…advantages in terms of statistical power, e.g., as already exploited in prior machine learn-1 ing work such as multi-task learning algorithms (Caruana, 1995; Baxter, 1997; Col-lobert and Weston, 2008) and learning algorithms involving parameter sharing (Lang and Hinton, 1988; LeCun, 1989).”
The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence (2020, background): “LeCun’s seminal work on convolution (LeCun, 1989) was in fact a strong empirical demonstration of precisely this, showing how learning in a digit recognition task, accuracy was better in a system that prewired hierarchical structure equipped with shift-invariant feature detectors (using…”
Representation Learning: A Review and New Perspectives (2012, background): “This allows pooling units to be invariant to more than just translations, and is a hybrid between convolutional networks and earlier neural networks with local connections but no weight sharing (LeCun, 1986, 1989).”
Explaining nonlinear classification decisions with deep Taylor decomposition (2015, background): “and that is frequently used for image recognition tasks, is the convolutional neural network [47].”
Deep Learning: A Critical Appraisal (2018, methodology): “Most deep learning networks make heavy use of a technique called convolution (LeCun, 1989), which constrains the neural connections in the network such that they innately capture a property known as translational invariance.”
Practical Recommendations for Gradient-Based Training of Deep Architectures (2012, methodology): “Multi-Task Learning and Multi-Relational Learning Parameter sharing (Lang and Hinton, 1988; LeCun, 1989; Lang and Hinton, 1988; Caruana, 1993; Baxter, 1995, 1997) be decomposed as the sum of a bias term and a variance term (Geman et al. , 1992).”
Recurrent Convolutional Neural Networks for Scene Parsing (2013, background): “Convolutional neural networks [11] are a natural extension of neural networks for treating images.”
Transformation Invariance in Pattern Recognition-Tangent Distance and Tangent Propagation (1996, background): “The network has two hidden layers with locally connected shared weights, and one output layer with 10 units (5194 connections, 1060 free parameters) [19].”
Estimating phoneme class conditional probabilities from raw speech signal using convolutional neural networks (2013, methodology): “Using MFCC features with the CNN architecture yields similar performance as the raw features.”
Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes (2018, background): “Locally Connected Networks (LCNs) (Fukushima, 1975; Lecun, 1989) are CNNs without weight sharing between spatial locations.”
Analysis of CNN-based speech recognition system using raw speech as input (2015, background): “In our recent studies [16, 17], it was shown that it is possible to estimate phoneme class conditional probabilities by using raw speech signal as input to convolutional neural networks [18] (CNNs).”
Deep Learning for Time Series Forecasting: Tutorial and Literature Survey (2020, background): “Convolutional neural networks (CNNs) [112] are a special class of NNs that are designed for applications where inputs have a known ordinal structure such as images and time series [70].”
First- and Second-Order Methods for Learning: Between Steepest Descent and Newton's Method (1992, methodology): “…maintains the "safety" properties even when the search is executed in a small number of one-dimensional trials.= While the above method is suitable for batch learning, a proposal for a learning method usable in the on-line procedure has been presented in LeCun (1989) and Becker and LeCun (1989).”
Graph convolutional networks for computational drug development and discovery (2019, background): “CNN reveals the state-of-art performance in image-related tasks since it can automatically extract task relevant features from raw images via a convolution operator [8].”
Universal Approximations of Invariant Maps by Neural Networks (2018, background): “[1989], le Cun [1989]) can be seen as a manifestation of the intrinsic approach (since the translational symmetry is built into the architecture of the network from the outset), and convnets are ubiquitous in modern machine learning LeCun et al. [2015]. In this paper we will be interested in the the”
Locality defeats the curse of dimensionality in convolutional teacher–student scenarios (2021, background): “Such functions model the local nature of certain datasets and can be generated, for example, by randomly-initialised one-hidden-layer neural networks: f LC corresponds to a locally connected network (LCN) [10, 11], in which the input is split into lower-dimensional patches before being processed, wh”
Convolutional face finder: a neural architecture for fast and robust face detection (2004, background): “[15], [28], [29], are powerful bioinspired hierarchical multilayered neural networks that combine three architectural ideas to ensure some degree of shift, scale, and distortion invariance: local receptive fields, shared weights, and spatial subsampling.”
Kernel Analysis of Deep Networks (2011, background): “Both approaches have shown empirically to produce better generalization (LeCun, 1989; Salakhutdinov and Hinton, 2007).”
Transformation invariance in pattern recognition: Tangent distance and propagation (2000, background): “The network has two hidden layers with locally connected shared weights and one output layer with 10 units (5,194 connections, 1,060 free parameters; LeCun, 1989).”