Learning Invariant Representations with Local Transformations

By Kihyuk Sohn, Honglak Lee. 2012

196 citations recorded by the source corpus.

Learning invariant representations is an important problem in machine learning and pattern recognition. In this paper, we present a novel framework of transformation-invariant feature learning by incorporating linear transformations into the feature learning algorithms. For example, we present the transformation-invariant restricted Boltzmann machine that compactly represents data by its weights and their transformations, which achieves invariance of the feature representation via probabilistic max pooling. In addition, we show that our transformation-invariant feature learning framework can also be extended to other unsupervised learning methods, such as autoencoders or sparse coding. We evaluate our method on several image classification benchmark datasets, such as MNIST variations, CIFAR-10, and STL-10, and show competitive or superior classification performance when compared to the state-of-the-art. Furthermore, our method achieves state-of-the-art performance on phone classification tasks with the TIMIT dataset, which demonstrates wide applicability of our proposed algorithms to other domains.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Signal Processing

Citations in the corpus

Spatial Transformer Networks (2015, background): “In this section we discuss the prior work related to the paper, covering the central ideas of modelling transformations with neural networks [12, 13, 27], learning and analysing transformation-invariant representations [3, 5, 8, 17, 19, 25], as well as attention and detection mechanisms for feature ”
Invariant Information Clustering for Unsupervised Image Classification and Segmentation (2018, methodology): “Optimising for function outputs to be persistent through spatio-temporal or non-material distortion is an idea shared by IIC with several works, including exemplars [18], IMSAT [28], proximity prediction [30], the denoising objective of Tagger [22], temporal slowness constraints [55], and optimising”
Deformable Convolutional Networks (2017, background): “Some works learn invariant CNN representations with respect to different types of transformations such as [50], scattering networks [3], convolutional jungles [32], and TI-pooling [33].”
Learning Image Representations Tied to Ego-Motion (2015, methodology): “Most relevant to our work are feature learning methods based on temporal coherence and “slow feature analysis” [32, 10, 22].”
A Hybrid Differential Evolution Approach to Designing Deep Convolutional Neural Networks for Image Classification (2018, neutral): “To be speciﬁc, the proposed DECNN ranks the ﬁfth on both the CONVEX and MB benchmark datasets; for the MBI benchmark, DECNN beats all of the state-of-the-art methods; for the MDRBI dataset, the mean error rate of DECNN is the fourth best, but the P-value of One Sample T-Test between DECNN and the th”
Conditional variance penalties and domain shift robustness (2017, background): “invariances from data and the focus often lies on geometric transformations of the input data or explicitly creating augmented observations (Sohn and Lee, 2012; Khasanova and Frossard, 2017; Hashimoto et al., 2017; Devries and Taylor, 2017).”
Group Equivariant Convolutional Networks (2016, methodology): “This includes work on transforming autoencoders (Hinton et al., 2011), equivariant Boltzmann machines (Kivinen & Williams, 2011; Sohn & Lee, 2012), equivariant descriptors (Schmidt & Roth, 2012), and equivariant ﬁltering (Skibbe, 2013).”
Harmonic Networks: Deep Translation and Rotation Equivariance (2016, neutral): “11 - Transformation RBM [25] 4 .”
Understanding Image Representations by Measuring Their Equivariance and Equivalence (2014, background): “A number of authors have looked at incorporating equivariance explicitly in the representations (Schimdt and Roth 2012a; Sohn and Lee 2012).”
Convolutional Kernel Networks (2014, neutral): “Method [12] [27] [18] [13] [4] [17] [31] CKN-GM CKN-PM CKN-CO CIFAR-10 82.”
Exploiting Cyclic Symmetry in Convolutional Neural Networks (2016, background): “Kivinen & Williams (2011), Sohn & Lee (2012) and Schmidt & Roth (2012) propose modiﬁed restricted Boltzmann machines (RBMs) with tied weights to achieve rotation equivariance.”
Manitest: Are classifiers really invariant? (2015, background): “Another approach for measuring invariance consists in generating datasets with transformed images, and measuring the accuracy of classiﬁers on these datasets [19, 21, 31].”
Deep roto-translation scattering for object classification (2014, neutral): “Dict 82.9 LIFT [27] Unsup.”
Learning Steerable Filters for Rotation Equivariant CNNs (2017, background): “Invariance to specific transformations can be achieved by applying them to the input and subsequently pooling their responses [11, 12, 13].”
Locally Scale-Invariant Convolutional Neural Networks (2014, methodology): “Method Test Error (%) on 6 train/test fold Restricted Boltzman Machine (RBM)[6] 6.”
Rotation Equivariant Vector Field Networks (2016, methodology): “The advantage of representing z in Cartesian coordinates is that the horizontal and vertical components [u, v] are orthogonal, and thus a convolution of the two vector fields can be computed on each component independently using standard convolutions (see Eq.”
TI-POOLING: Transformation-Invariant Pooling for Feature Learning in Convolutional Neural Networks (2016, methodology): “The best approach by [25] employs restricted boltzmann machines and achieves 4.”
Oriented Response Networks (2017, neutral): “ScatNet-2 [3] 7.48 PCANet-2 [4] 7.37 TIRBM [40] 4.2 CNN 4.34 ORN-8(ORAlign) 2.25 TIPooling(with augmentation) [23] 1.93 OR-TIPooling(with augmentation) 1.54 An interesting experiment comes from the digit class ‘6’ and ‘9’.”
Local Group Invariant Representations via Orbit Embeddings (2016, methodology): “The proposed LGIKA outperforms most of the competing methods including deep architectures like rotation-invariant convolutional RBM (RC-RBM) [40], transformation invariant RBM (TI-RBM) [43], and regular deep CNN (Z2-CNN) [11].”