Hypercolumns for object segmentation and fine-grained localization

By Bharath Hariharan, Pablo Arbeláez, Ross B. Girshick, Jitendra Malik. 2014

1,647 citations recorded by the source corpus.

Recognition algorithms based on convolutional networks (CNNs) typically use the output of the last layer as a feature representation. However, the information in this layer may be too coarse spatially to allow precise localization. On the contrary, earlier layers may be precise in localization but will not capture semantics. To get the best of both worlds, we define the hypercolumn at a pixel as the vector of activations of all CNN units above that pixel. Using hypercolumns as pixel descriptors, we show results on three fine-grained localization tasks: simultaneous detection and segmentation [22], where we improve state-of-the-art from 49.7 mean APr [22] to 60.0, keypoint localization, where we get a 3.3 point boost over [20], and part labeling, where we show a 6.6 point gain over a strong baseline.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Universal Language Model Fine-tuning for Text Classification (2018, methodology): “This method is known as hypercolumns (Hariharan et al., 2015) in CV and is used by Peters et al. (2017), Peters et al. (2018), Wieting and Gimpel (2017), Conneau et al. (2017), and McCann et al. (2017) who use language modeling, paraphrasing, entailment, and Machine Translation (MT) respectively…”
U-Net: Convolutional Networks for Biomedical Image Segmentation (2015, background): “More recent approaches [11,4] proposed a classifier output that takes into account the features from multiple layers.”
Mask R-CNN (2017, methodology): “Earlier methods [13, 15, 16, 9] resorted to bottom-up segments [38, 2].”
Fine-tuned Language Models for Text Classification (2018, methodology): “This approach is known as hypercolumns in CV (Hariharan et al., 2015) and is used by Peters et al. (2017), Anonymous (2018), and McCann et al. (2017) who use language modeling and Machine Translation (MT) respectively for pretraining.”
Growing a Brain: Fine-Tuning by Increasing Model Capacity (2017, background): “Fine-tuning: Fine-tuning is by far the dominant strategy for transfer learning with neural networks [28, 4, 32, 53, 10, 12].”
Learning Features by Watching Objects Move (2016, background): “ConvNet-based image representations are extremely versatile, showing good performance in a variety of recognition tasks [9, 15, 19, 50].”
Feature Pyramid Networks for Object Detection (2016, methodology): “Hypercolumns [13] uses a similar method for object instance segmentation.”
Colorful Image Colorization (2016, methodology): “use hypercolumns [25] on a VGG network [5], Iizuka et al.”
UPSNet: A Unified Panoptic Segmentation Network (2019, background): “Recent semantic segmentation methods that exploit deep convolutional feature extraction mainly approach this problem from either multi-scale feature aggregation [40, 26, 5, 14], or end-to-end structured prediction [2, 41, 1, 25, 4] perspectives.”
Recurrent Neural Networks for Semantic Instance Segmentation (2017, methodology): “In this paper, we argue that both ranking thousands of object-prominent regions [17, 18, 24, 23, 11, 19] or decomposing the problem into two independent modules [6, 12, 34] are unnatural approaches that are far away from the sequential exploration that we as humans follow to solve related tasks such”
Fully Convolutional Instance-Aware Semantic Segmentation (2016, methodology): “In a prevalent family of instance-aware semantic segmentation approaches [7, 16, 8], it is achieved by adopting different types of sub-networks in three stages: 1) an FCN is applied on the whole image to generate intermediate and shared feature maps; 2) from the shared feature maps, a pooling layer ”
SSD: Single Shot MultiBox Detector (2015, methodology): “[14, 7] have shown that using feature maps from the lower layers can improve semantic segmentation quality because the lower layers capture more fine details of the input objects.”
Pixelwise Instance Segmentation with a Dynamically Instantiated Network (2017, methodology): “Furthermore, since we are considering object detection outputs jointly with semantic segmentation predictions, we have some robustness to high-scoring false positive detections unlike methods such as [7, 19, 32] which refine object detections into segmentations.”
Beyond Skip Connections: Top-Down Modulation for Object Detection (2016, methodology): “For example, [4, 10, 22, 42] combine features from multiple layers for the final classifier; while [4, 42] use subsampled features from finer scales, [10, 22] upsample the features to the finest scale and use their combination.”
Shape-aware Instance Segmentation (2016, methodology): “These baselines include the SDS framework of [13], the Hypercolumn representation of [14], the InstanceFCN method of [5] and the MNC framework of [7].”
InstanceCut: From Edges to Instances with MultiCut (2016, methodology): “Similarly constructed features also have been used successfully for other dense labelling problems [23].”
The Cityscapes Dataset for Semantic Urban Scene Understanding (2016, methodology): “Other work takes advantage of deep learning for explicitly integrating global scene context in the prediction 5 of pixel-wise semantic labels, in particular through CNNs [4,39,44,67] or Recurrent Neural Networks (RNNs) [8,52].”
Instance-Sensitive Fully Convolutional Networks (2016, methodology): “Following [3,4], we use the segmentation annotations from [27], and train the models on the training set and evaluate on the validation set.”
Learning to Refine Object Segments (2016, methodology): “2 in [24], Fig.”