Tangent Prop - A Formalism for Specifying Selected Invariances in an Adaptive Network

By Patrice Y. Simard, Bernard Victorri, Yann LeCun, John S. Denker. 1991

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

Citations in the corpus

A Generative Process for Contractive Auto-Encoders (2012, methodology): “tangent" directions in a subsequent supervised classication training to encourage class prediction probabilities to be invariant to these directions, by using the tangent-propagation technique (Simard et al., 1992). A Generative Process for Sampling Contractive Auto-Encoders 3. Generating sampl”
Semi-supervised Learning with Deep Generative Models (2014, methodology): “Currently, state-of-the-art results on a semi-supervised version of the MNIST benchmark data set have been achieved by the Manifold Tangent Classifier (MTC) (Rifai et al., 2011), which trains contrastive auto-encoders (CAEs) to learn the manifold on which the data lies, followed by an instance of Ta”
Representation Learning: A Review and New Perspectives (2012, methodology): “…Tangent Classifier (MTC), Rifai et al. (2011c) then apply techniques such as tangent distance (Simard et al., 1993) and tangent propagation (Simard et al., 1992), that were initially developed to build classifiers that are insensitive to input deformations provided as prior domain…”
Toward Causal Representation Learning (2021, background): “So far, there has been no definitive consensus on how to solve these problems, although progress has been made using data augmentation, pretraining, self-supervision, and architectures with suitable inductive biases with respect to a perturbation of interest [60], [64], [137], [170], [206], [233].”
Data Augmentation Generative Adversarial Networks (2017, background): “This relates to the idea of tangent distances (Simard et al., 1991), but involves targeting the distances between manifolds deﬁned by a learnt DAGAN.”
New Insights and Perspectives on the Natural Gradient Method (2014, methodology): “To compute such products efficiently one can use the method of Schraudolph (2002), which is a generalization of the well-known method for computing such products with the classical Gauss-Newton (and is also related to the TangentProp method of Simard et al. (1992)).”
APAC: Augmented PAttern Classification with Neural Networks (2015, background): “Tangent Prop [16] is a way to avoid over-ﬁtting with implicit use of data augmentation.”
CubeNet: Equivariance to 3D Rotation and Translation (2018, background): “There are also examples of CNNs, which have explicit regularization to learn equivariance [43, 30, 51, 40].”
Sequence Learning (2001, background): “One possibility is to give some hints to the network on specific properties of the desired function (see for example [1] [6] [58] [3] [2] [5] [4]) or to insert prior knowledge in form of rules into the neural network and then to train it using a standard learning algorithm (for the case of dynamical”
Further Advantages of Data Augmentation on Convolutional Neural Networks (2018, methodology): “Data augmentation was already used in the late 80’s and early 90’s for handwritten digit recognition Simard et al. (1992) and it has been identiﬁed as a very important element of many modern successful models, like AlexNet Krizhevsky et al. (2012), All-CNN Springenberg et al. (2014) or ResNet He et…”
A Group-Theoretic Framework for Data Augmentation (2019, background): “Starting from [30, 45, 61, 14], a line of research focuses on designing (or learning) neural net architectures that are invariant to certain symmetries [31, 27, 70, 20, 22, 19], equivariant to SO(3) group [21, 28, 29], and others [58, 42, 69, 41].”
State-Denoised Recurrent Neural Networks (2018, background): “More recently in machine learning, loss functions have been explored to achieve invariance to task-irrelevant perturbations in the input [Simard et al., 1992, Zheng et al., 2016].”
Ieee Transactions on Pattern Analysis and Machine Intelligence 1 Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networks (2014, unknown): “The research most similar to ours is early work on tangent propagation [17] (and the related double backpropagation [18]) which aims to learn invariance to small predefined transformations in a neural network by directly penalizing the derivative of the output with respect to the magnitude of the…”