Geodesics of learned representations

By Olivier J. H'enaff, Eero P. Simoncelli. 2015

55 citations recorded by the source corpus.

We develop a new method for visualizing and refining the invariances of learned representations. Specifically, we test for a general form of invariance, linearization, in which the action of a transformation is confined to a low-dimensional subspace. Given two reference images (typically, differing by some transformation), we synthesize a sequence of images lying on a path between them that is of minimal length in the space of the representation (a "representational geodesic"). If the transformation relating the two reference images is linearized by the representation, this sequence should follow the gradual evolution of this transformation. We use this method to assess the invariance properties of a state-of-the-art image classification network and find that geodesics generated for image pairs differing by translation, rotation, and dilation do not evolve according to their associated transformations. Our method also suggests a remedy for these failures, and following this prescription, we show that the modified representation is able to linearize a variety of geometric image transformations.

Subfields: Artificial Intelligence, Biophysics, Media Technology, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biophysics

In Media Technology

In Representation & Self-Supervised Learning

Citations in the corpus

Making Convolutional Networks Shift-Invariant Again (2019, methodology): “While previous work (Scherer et al., 2010; H´enaff & Simoncelli, 2016; Azulay & Weiss, 2019) acknowledges the drawbacks of max-pooling and beneﬁts of blurred-downsampling, they are viewed as separate, discrete choices, preventing their combination.”
Why do deep convolutional networks generalize so poorly to small image transformations? (2018, methodology): “method to probe the invariances measures the linearity of the learned representations under natural transformations to the input image [28, 22, 16, 9].”
E-LPIPS: Robust Perceptual Image Similarity via Random Transformation Ensembles (2019, background): “Prior study has hinted toward similar interesting properties of the discrete geodesics of the “VGG feature” image metric: Hénaff and Simoncelli [10] ﬁnd evidence of linearization of effects such as small translation and rotation, but only when aided with additional projections into pixel-space…”
Image Quality Assessment: Unifying Structure and Texture Similarity (2020, methodology): “Following this principle, we replace all max pooling layers in VGG with weighted (cid:96) 2 pooling [43]: where (cid:12) denotes pointwise product, and the blurring kernel g ( · ) is implemented by a Hanning window that approximately enforces the Nyquist criterion.”
DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data (2023, background): “Further improvements include ensembling for robustness [45], antialiasing for stability [24, 36, 93], and global descriptors for texture [24].”
MAKING CONVOLUTIONAL NETWORKS SHIFT-INVARIANT AGAIN (2018, background): “…activate hidden units (Girshick et al., 2014; Zhou et al., 2014), actively maximizing hidden units (Mordvintsev et al., 2015), and mapping features back into pixel space (Dosovitskiy & Brox, 2016a;b; Mahendran & Vedaldi, 2015; Zeiler & Fergus, 2014; Nguyen et al., 2017; Hénaff & Simoncelli, 2015).”
No-Reference Image Quality Assessment via Transformers, Relative Ranking, and Self-Consistency (2021, methodology): “For normalization and pooling we use Euclidean norm which is defined by Fi = Fi max(‖Fi‖2, ) followed by a l2 pooling layer [69, 70] which has been used to demonstrate the behavior of complex cells in primary visual cortex [71, 72].”
Eigen-Distortions of Hierarchical Representations (2017, unknown): “In their paper, Johnson et al. [2016] trained a neural network to generate super-resolution images using the representation of an intermediate layer of VGG16 as a perceptual loss function, and showed that the images this network produced looked significantly better than images generated with simpler”