Do Convnets Learn Correspondence?

By Jonathan Long, Ning Zhang, Trevor Darrell. 2014

316 citations recorded by the source corpus.

Convolutional neural nets (convnets) trained from massive labeled datasets [1] have substantially improved the state-of-the-art in image classification [2] and object detection [3]. However, visual understanding requires establishing correspondence on a finer level than object category. Given their large pooling regions and training from whole-image labels, it is not clear that convnets derive their success from an accurate correspondence model which could be used for precise localization. In this paper, we study the effectiveness of convnet activation features for tasks requiring correspondence. We present evidence that convnet features localize at a much finer scale than their receptive field sizes, that they can be used to perform intraclass aligment as well as conventional hand-engineered features, and that they outperform conventional features in keypoint prediction on objects from PASCAL VOC 2011 [4].

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Fully convolutional networks for semantic segmentation (2014, background): “These include advances in bounding box object detection [29, 10, 17], part and keypoint prediction [39, 24], and local correspondence [24, 8].”
Deformable part models are convolutional neural networks (2014, positive): “This observation confirms recent observations in [30].”
Generative Modeling of Convolutional Neural Networks (2014, methodology): “We sample from the well-defined probability distribution by the HMC algorithm, generating meaningful and varying synthetic images, without resorting to a large collection of hold-out images [6, 26, 19].”
Viewpoints and keypoints (2014, methodology): “Our baseline system ( ’conv6+conv12’) performs much better than [25], indicating the importance of end-toend training and multiscale response maps.”
Dense Human Body Correspondences Using Convolutional Networks (2015, background): “Several recent papers [27, 15, 49, 14] have successfully used this idea for finding correspondences between 2D images, and have shown that descriptors learned from deep neural networks are significantly better than generic pixel-wise descriptors in this context.”
Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes (2018, background): “(1)While there is broad literature on empirical interpretation of finite CNNs (Zeiler & Fergus, 2014; Simonyan et al., 2014; Long et al., 2014; Olah et al., 2017), it is commonly only applicable to fully trained networks.”
Slice-to-volume medical image registration: a survey (2017, background): “More recently, features learned using deep learning have been successfully applied for scenarios involving massive amount of annotated data for training (Long et al., 2014).”
WarpNet: Weakly Supervised Matching for Single-View Reconstruction (2016, methodology): “with weights pre-trained on the ILSVRC classification task, following prior state-of-the-art for correspondence [19].”
StarMap for Category-Agnostic Keypoint and Viewpoint Estimation (2018, methodology): “1) is marginally better than the state-of-the-arts in 2014 [16,37], probably because we used a more up-to-date HourglassNetwork [24].”
Understanding Intra-Class Knowledge Inside CNN (2015, methodology): “As discovered in [7], conv-layer features can be used to retrieve image patches with similar appearance, though their semantic can be totally different.”
Aggregating Local Deep Features for Image Retrieval (2015, methodology): “Perhaps inspired by this analogy, [15] suggested to use such features to identify meaningful object parts, while [5] proposed to use Fisher vector [23] constructed on these local features to produce a global image descriptor that provides state-of-the-art classification performance on external datas”
Visualizations of Deep Neural Networks in Computer Vision: A Survey (2017, background): “Long et al.[45] 2014 Architecture assessment Pixel displays Classification CNNs ImageNet, VOC”
6-DoF object pose from semantic keypoints (2017, background): “, [22], [23], [24], [25]) has attracted considerable study, while limited attention has been given to their application with generic object categories [26], [20].”
Unsupervised learning of object semantic parts from internal states of CNNs by population encoding (2015, methodology): “Note that this baseline is very similar to the keypoint prediction method in [12] but with a different performance measure.”
I Have Seen Enough: Transferring Parts Across Categories (2016, background): “As common practice [31, 61], keypoint detections are restricted to ground-truth object bounding boxes for all evaluation measures.”
Finding and Visualizing Weaknesses of Deep Reinforcement Learning Agents (2019, background): “[25] reconstruct an image from an average of image patches based on nearest neighbors in feature space.”
Towards Interpretable Object Detection by Unfolding Latent Structures (2019, methodology): “There are a variety of methods including identifying high-scoring image patches [20, 43] or over-segmented atomic regions [51] directly, visualizing the layers of convolutional networks using deconvolutional networks to understand what contents are emphasized in the high-scoring input image patches ”