Learning Transferrable Knowledge for Semantic Segmentation with Deep Convolutional Neural Network

By Seunghoon Hong, Junhyuk Oh, Honglak Lee, Bohyung Han. 2015

177 citations recorded by the source corpus.

We propose a novel weakly-supervised semantic segmentation algorithm based on Deep Convolutional Neural Network (DCNN). Contrary to existing weakly-supervised approaches, our algorithm exploits auxiliary segmentation annotations available for different categories to guide segmentations on images with only image-level class labels. To make segmentation knowledge transferrable across categories, we design a decoupled encoder-decoder architecture with attention model. In this architecture, the model generates spatial highlights of each category presented in images using an attention model, and subsequently performs binary segmentation for each highlighted region using decoder. Combining attention model, the decoder trained with segmentation annotations in different categories boosts accuracy of weakly-supervised semantic segmentation. The proposed algorithm demonstrates substantially improved performance compared to the state-of-theart weakly-supervised techniques in PASCAL VOC 2012 dataset when our model is trained with the annotations in 60 exclusive categories in Microsoft COCO dataset.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Fully Convolutional Instance-Aware Semantic Segmentation (2016, background): “As the per-pixel category labeling is expensive, the supervision signals in FCNs have been relaxed to boxes [6], scribbles [23], or weakly supervised image class labels [19, 20].”
Representation Compensation Networks for Continual Semantic Segmentation (2022, methodology): “Some methods [14,22,32,37,38,50,55] is inspired by Non-local [85], utilizing attention mechanisms to establish connections between image contexts.”
ABCNN: Attention-Based Convolutional Neural Network for Modeling Sentence Pairs (2015, background): “, 2015), image segmentation (Hong et al., 2016) and object localization (Cao et al.”
Large-Scale Weakly-Supervised Pre-Training for Video Action Recognition (2019, background): “Learning from weak supervision: Given the known challenges in gathering exhaustive annotations for various image and video tasks, leveraging object labels and other meta information to supply weak supervision [7, 15, 19, 22, 30, 31, 39, 40, 50, 52, 53, 54, 55, 56, 59] is a widely-adopted approach.”
Strip Pooling: Rethinking Spatial Pooling for Scene Parsing (2020, background): “One way to improve the capability of modeling the long-range dependencies in CNNs is to adopt self-attention or non-local modules [51, 23, 7, 45, 21, 53, 66, 62, 61, 28].”
FCNs in the Wild: Pixel-level Adversarial and Constraint-based Adaptation (2016, background): “In another recent paper [15], Hong et al . used auxiliary segmentation to generalize semantic segmentations to categories where only weak label information was available.”
One-Shot Learning for Semantic Segmentation (2017, background): “ned for different tasks or objects [34]. In contrast, we meta-learn a network to produce a high-level representation of a new semantic class given a single labeled example. Semi-supervised approaches [18,19,30] combine weak labels with a small set of pixel-level annotations. However, they assume a l”
Weakly Supervised Semantic Segmentation Using Web-Crawled Videos (2017, methodology): “However, they often require additional human intervention to obtain extra supervision [6, 13, 26] or employ domainspecific knowledge that may not be well-generalized to other classes [28].”
Customizing Student Networks From Heterogeneous Teachers via Adaptive Knowledge Amalgamation (2019, background): “It contains two main research directions: cross-domain transfer learning [22, 12, 10, 4] and cross-task one [9, 3, 5, 35].”
Sidekick Policy Learning for Active Visual Exploration (2018, background): “Methods for depth estimation [22,16,60] and/or semantic segmentation [56,25,26] use RGBD depth data, multiple views, and/or auxiliary annotations during training, then proceed with single image observations at test time.”
Deep Visual Domain Adaptation: A Survey (2018, methodology): “[44] used a novel encoder-decoder architecture with attention model by transferring weak class labeled knowledge in the source domain, while [51, 81] transferred weak object location knowledge.”
Learning Pixel-Level Semantic Affinity with Image-Level Supervision for Weakly Supervised Semantic Segmentation (2018, methodology): “Extra Data val test TransferNet [10] I MS-COCO [20] 52.”
Optimizing Intersection-Over-Union in Deep Neural Networks for Image Segmentation (2016, background): “Very recently, some deep encoder-decoder based models (e.g., SegNet [16], TransferNet [17]) have been proposed that mark the state-of-the-art for image semantic segmentation.”
Weakly-Supervised Semantic Segmentation Network with Deep Seeded Region Growing (2018, methodology): “Among the approaches, CCNN [21], MIL-seg [23], STC [35], GuidedSeg [19], and TransferNet [10] use more images for training (700K, 700K, 50K, 20K and 70K, respectively).”
Revisiting Dilated Convolution: A Simple Approach for Weakly- and Semi-Supervised Semantic Segmentation (2018, unknown): “MIL-* [24] (700K), TransferNet [7] (70K), STC [34] (50K) and Hong et al .”
FickleNet: Weakly and Semi-Supervised Semantic Image Segmentation Using Stochastic Inference (2019, methodology): “These methods include TransferNet [9], which was trained on pixel-level annotations of 60 classes (not Pascal VOC classes) of COCO [20] images, and CrawlSeg [10], which Table 3.”
PointGrid: A Deep Network for 3D Shape Understanding (2018, methodology): “Deep learning has become a universal tool for many visual recognition tasks ranging from classification to segmentation, especially ConvNets for 2D images [28, 48, 50, 16, 10, 34, 37, 45, 18] thanks to its weight sharing and other kernel optimizations of 2D convolutions.”
Seed, Expand and Constrain: Three Principles for Weakly-Supervised Image Segmentation (2016, unknown): “2∗ trained on Flickr TransferNet [12] 52.”
The Mapillary Vistas Dataset for Semantic Understanding of Street Scenes (2017, background): “Further works have focused on relaxing the level of supervision in FCNs by considering bounding boxes [7], scribbles [27], or image class labels [20, 21] as sources of weak supervision.”