On Invariance and Selectivity in Representation Learning

By Fabio Anselmi, Lorenzo Rosasco, Tomaso A. Poggio. 2015

110 citations recorded by the source corpus.

AbstractWe discuss data representation which can be learned automaticallyfrom data, are invariant to transformations, and at the same time selec-tive, in the sense that two points have the same representation only if theyare one the transformation of the other. The mathematical results heresharpen some of the key claims of i-theory { a recent theory of feedforwardprocessing in sensory cortex. [3, 4, 5].Keywords:Invariance, Machine LearningThe paper is submitted to Information and Inference Journal. 1 Introduction This paper considers the problem of learning "good" data representation whichcan lower the need of labeled data (sample complexity) in machine learning(ML). Indeed, while current ML systems have achieved impressive results in avariety of tasks, an obvious bottleneck appears to be the huge amount of labeleddata needed. This paper builds on the idea that data representation, which arelearned in an unsupervised manner, can be key to solve the problem. Classicalstatistical learning theory focuses on supervised learning and postulates that asuitable hypothesis space is given. In turn, under very general conditions, thelatter can be seen to be equivalent to a data representation. In other words,data representation and how to select and learn it, is classically not consideredto be part of the learning problem, but rather as a prior information. In practicead hoc solutions are often empirically found for each problem.The study in this paper is a step towards developing a theory of learningdata representation. Our starting point is the intuition that, since many learning

Subfields: Artificial Intelligence, Cognitive Neuroscience, Computational Theory and Mathematics, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Computational Theory and Mathematics

In Representation & Self-Supervised Learning

Citations in the corpus

GSNs : Generative Stochastic Networks (2015, background): “See also the work on learning invariant representations in this issue (Anselmi et al., 2016; Cheng et al., 2016).”
View-tolerant face recognition and Hebbian learning imply mirror-symmetric neural tuning to head orientation (2016, positive): “Our assumptions about the architecture for invariance conform to i-theory [4, 5] which is a theory of invariant recognition that characterizes and generalizes the convolutional and pooling layers in deep networks.”
Towards a Definition of Disentangled Representations (2018, methodology): “Group theory also forms the foundation of Anselmi et al’s theory of visual recognition with hierarchical models [8].”
Variational Inference of Disentangled Latent Concepts from Unlabeled Observations (2017, background): “…knowledge about the nature of input transformations (e.g., rotations, translations, scaling for images, etc.) (Schmidt and Roth, 2012; Bruna and Mallat, 2013; Anselmi et al., 2014, 2016; Cohen and Welling, 2016; Dieleman et al., 2016; Haasdonk et al., 2005; Mroueh et al., 2015; Raj et al., 2017).”
Information Dropout: Learning Optimal Representations Through Noisy Computation (2016, background): “In these cases, which cover common nuisances such as translations and rotations of an image (affine group), or small diffeomorphic deformations due to a slight change of point of view (group of diffeomorphisms), the representation is equivalent to the data modulo the action of the group.”
Emergence of Invariance and Disentanglement in Deep Representations (2017, background): “Anselmi et al. (2016) develop a theory of invariance to locally compact groups, and aim to construct maximal (“distinctive”) invariants, like Sundaramoorthi et al. (2009) that, however, assume nuisances to be infinitedimensional groups (Grenander, 1993). These efforts are limited by the assumption t”
Group Invariance, Stability to Deformations, and Complexity of Deep Convolutional Representations (2017, unknown): “…for more classical convolutional neural networks from methodological and empirical points of view (Bruna et al., 2013; Cohen and Welling, 2016), and for shallow learned representations (Anselmi et al., 2016) or kernel methods (Haasdonk and Burkhardt, 2007; Mroueh et al., 2015; Raj et al., 2017).”
Why neurons mix: high dimensionality for higher cognition. (2016, background): “The resulting representation clearly allows for a potentially large reduction in sample complexity, since the label provided by a training sample can be generalized over all the inputs corresponding to the same object [26,28].”
Invariance and Stability of Deep Convolutional Representations (2017, unknown): “Invariance to groups of transformations was also studied for more classical convolutional neural networks from methodological and empirical points of view [6, 9], and for shallow learned representations [1] or kernel methods [13, 19, 22].”
Information Dropout: learning optimal representations through noise (2017, methodology): “…inﬁnite-dimensional) group acting on the data, like Sundaramoorthi et al. (2009), or, when the nuisance is a locally compact group acting on each layer, by successive approximations implemented by hierarchical convolutional architectures, like Anselmi et al. (2016) and Bruna & Mallat (2011).”
Discriminative template learning in group-convolutional networks for invariant speech representations (2015, background): “Representations invariant to transformations [7] that have a locally compact group structure can be obtained through a feature map that defines equivalence classes [8, 9].”
Emergence of invariance and disentangling in deep representations (2017, background): “For instance, the use of convolutional layers and max-pooling can be shown to yield insensitivity to local group transformations (Bruna & Mallat, 2011; Anselmi et al., 2016; Soatto & Chiuso, 2016).”
Generalization Error of Invariant Classifiers (2016, background): “Poggio et al. (2012) and Anselmi et al. (2014, 2016) study biologically plausible learning of invariant representations and connect their results to CNNs.”
Visual Representations: Defining Properties and Deep Approximations (2014, methodology): “In particular, Anselmi et al. (2015) define a G-invariant representation µ of a (single) image I as being selective if µ(I) = µ(I ′) ⇒ I ∼ I ′ for all I, I ′, i.e., if it is a maximal invariant.”
Deep Convolutional Networks are Hierarchical Kernel Machines (2015, methodology): “Then we establish some of their mathematical properties by using i-theory as described in [8, 9, 10].”
SE(3)-equivariant prediction of molecular wavefunctions and electronic densities (2021, background): “[59, 60]), increasing model performance and decreasing the required amount of training data.”