Fully convolutional networks for semantic segmentation
By Evan Shelhamer, Jonathan Long, Trevor Darrell. 2014
42,416 citations recorded by the source corpus.
Convolutional networks are powerful visual models that yield hierarchies of features. We show that convolutional networks by themselves, trained end-to-end, pixels-to-pixels, exceed the state-of-the-art in semantic segmentation. Our key insight is to build “fully convolutional” networks that take input of arbitrary size and produce correspondingly-sized output with efficient inference and learning. We define and detail the space of fully convolutional networks, explain their application to spatially dense prediction tasks, and draw connections to prior models. We adapt contemporary classification networks (AlexNet [20], the VGG net [31], and GoogLeNet [32]) into fully convolutional networks and transfer their learned representations by fine-tuning [3] to the segmentation task. We then define a skip architecture that combines semantic information from a deep, coarse layer with appearance information from a shallow, fine layer to produce accurate and detailed segmentations. Our fully convolutional network achieves state-of-the-art segmentation of PASCAL VOC (20% relative improvement to 62.2% mean IU on 2012), NYUDv2, and SIFT Flow, while inference takes less than one fifth of a second for a typical image.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Mohit Bansal
- Quoc V. Le
- Dacheng Tao
- Sergey Levine
- Jianfeng Gao
- Trevor Darrell
- Percy Liang
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Xiaogang Wang
- Trevor Darrell
- Jitendra Malik
- Yi Yang
- Andrew Zisserman
Citations in the corpus
- Deep Residual Learning for Image Recognition (2015, background): “Many other nontrivial visual recognition tasks [8, 12, 7, 32, 27] have also”
- Rethinking the Inception Architecture for Computer Vision (2015, background): “…[16] winning entry by Krizhevsky et al [9], their network “AlexNet” has been successfully applied to a larger variety of computer vision tasks, for example to object-detection [5], segmentation [12], human pose estimation [22], video classification [8], object tracking [23], and superresolution [3].”
- Universal Language Model Fine-tuning for Text Classification (2018, methodology): “For this reason, most work in CV focuses on transferring the last layers of the model (Long et al., 2015b).”
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks (2015, methodology): “Our RPNs are thus a kind of fully-convolutional network (FCN) [14] and they can be trained end-to-end specifically for the task for generating detection proposals.”
- Object Detection Networks on Convolutional Feature Maps (2015, methodology): “Then we adopt the “hole algorithm” [33], 464 [34] (“Algorithme à trous” [35]) on all following layers 465 to compensate this modification.”
- U-Net: Convolutional Networks for Biomedical Image Segmentation (2015, methodology): “In this paper, we build upon a more elegant architecture, the so-called “fully convolutional network” [9].”
- Mask R-CNN (2017, methodology): “The mask branch is a small FCN applied to each RoI, predicting a segmentation mask in a pixel-topixel manner.”
- Momentum Contrast for Unsupervised Visual Representation Learning (2019, methodology): “ImageNet supervised pre-training is most influential when serving as the initialization for fine-tuning in downstream tasks ( e.g ., [21, 20, 43, 52]).”
- Parameter-Efficient Transfer Learning for NLP (2019, methodology): “This technique attains state-of-the-art performance on many vision tasks, including classification (Kornblith et al., 2018), fine-grained classifcation (Hermans et al., 2017), segmentation (Long et al., 2015), and detection (Girshick et al., 2014).”
- Rethinking ImageNet Pre-Training (2018, background): “Pre-training has enabled state-of-the-art results on many tasks, including object detection [9, 8, 36], image segmentation [29, 13], and action recognition [42, 4].”
- Exploring the Limits of Weakly Supervised Pretraining (2018, background): “Examples include: object detection [1,4], semantic segmentation [5,6], human pose estimation [7,8], video recognition [9], monocular depth estimation [10], and so on.”
- Fine-tuned Language Models for Text Classification (2018, methodology): “For this reason, most work in CV focuses on transferring the last layers of the model ( Long et al., 2015b).”
- Exploring Self-Attention for Image Recognition (2020, background): “These serve as the basis for models developed for other computer vision tasks, such as semantic segmentation [22, 3, 42, 44] and object detection [10, 9, 28, 21].”
- Weight Standardization (2019, background): “Deep neural networks achieve state-of-the-arts in many vision tasks [4, 14, 20, 25, 31, 32, 39, 46].”
- CCNet: Criss-Cross Attention for Semantic Segmentation (2018, methodology): “For clear display, residual connections are ignored. the fully convolutional network (FCN) [26] have made remarkable progress.”
- Group Normalization (2018, background): “2017), segmentation (Long et al. 2015; He et al. 2017), video recognition (Tran et al.”
- Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (2020, background): “Learning good image representations is a key challenge in computer vision [1, 2, 3] as it allows for efficient training on downstream tasks [4, 5, 6, 7].”
- VirTex: Learning Visual Representations from Textual Annotations (2020, background): “This approach has been wildly successful, and has led to significant advances on a wide variety of computer vision problems such as object detection [7], semantic [8] and instance [9] segmentation, image captioning [10–12], and visual question answering [13, 14].”