Learning to Relate Images

By Roland Memisevic. 2011

72 citations recorded by the source corpus.

A fundamental operation in many vision tasks, including motion understanding, stereopsis, visual odometry, or invariant recognition, is establishing correspondences between images or between images and data from other modalities. Recently, there has been increasing interest in learning to infer correspondences from data using relational, spatiotemporal, and bilinear variants of deep learning methods. These methods use multiplicative interactions between pixels or between features to represent correlation patterns across multiple images. In this paper, we review the recent work on relational feature learning, and we provide an analysis of the role that multiplicative interactions play in learning to encode relations. We also discuss how square-pooling and complex cell models can be viewed as a way to represent multiplicative interactions and thereby as a way to encode relations.

Subfields: Biophysics, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biophysics

In Computer Vision and Pattern Recognition

Citations in the corpus

A Structured Self-attentive Sentence Embedding (2017, methodology): “Since here we are just computing the factor of relations ( F r ) from M h and M p , it corresponds to the encoder part in the Factored Gated Autoencoder in Memisevic (2013).”
Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images (2015, background): “Second, learning relations between pairs of images – although without control – has received considerable attention from the community during the last years [31, 32].”
Action-Conditional Video Prediction using Deep Networks in Atari Games (2015, background): “The advantages of multiplicative interactions have been explored in image and text processing [33, 30, 18].”
Unsupervised Learning of Video Representations using LSTMs (2015, background): “Generative models for understanding transformations between pairs of consecutive images are also well studied (Memisevic, 2013; Memisevic & Hinton, 2010; Susskind et al., 2011).”
Deep Kalman Filters (2015, methodology): “ndom noise that is this high, one needs to have a model of sequences capable of performing “ﬁltering”. Models that rely on predicting the next image based on the previous one (Goroshin et al. , 2015; Memisevic, 2013) may not sufﬁce to learn the structure of digits in the presence of large noise and ”
The “Something Something” Video Database for Learning and Evaluating Visual Common Sense (2017, background): “The fact that multiple images can reveal qualitatively different information on top of still images has been well-established for many years in both the computer vision and deep learning communities, and it has been exploited in the context of deep learning largely by using bilinear models (see [18]”
Slow and Steady Feature Analysis: Higher Order Temporal Coherence in Video (2015, neutral): “Another class of methods learns transformations [29, 28, 33].”
Modeling Deep Temporal Dependencies with Recurrent "Grammar Cells" (2014, methodology): “Due to the pooling layer, W, they are furthermore independent of the absolute angles in the subspaces [14].”
Learning Image Representations Tied to Egomotion from Unlabeled Video (2017, methodology): “In [35], bilinear models with multiplicative interactions are used to learn contentindependent “motion features” that encode only the transformation between image pairs.”
Learning image representations equivariant to ego-motion (2015, methodology): “In [18], bilinear models with multiplicative interactions are used to learn contentindependent " motion features " that encode only the transformation between image pairs.”
Unsupervised Learning of Visual Representations using Videos (2015, background): “Memisevic (2013) has a review of these related models.”
Learning Transferrable Knowledge for Semantic Segmentation with Deep Convolutional Neural Network (2015, methodology): “To obtain category-specific attention α using our attention model fatt, we employ multiplicative interactions [23] between feature and label vector.”