Multi-Scale Context Aggregation by Dilated Convolutions
By Fisher Yu, Vladlen Koltun. 2015
9,514 citations recorded by the source corpus.
State-of-the-art models for semantic segmentation are based on adaptations of convolutional networks that had originally been designed for image classification. However, dense prediction and image classification are structurally different. In this work, we develop a new convolutional network module that is specifically designed for dense prediction. The presented module uses dilated convolutions to systematically aggregate multi-scale contextual information without losing resolution. The architecture is based on the fact that dilated convolutions support exponential expansion of the receptive field without loss of resolution or coverage. We show that the presented context module increases the accuracy of state-of-the-art semantic segmentation systems. In addition, we examine the adaptation of image classification networks to dense prediction and show that simplifying the adapted network can increase accuracy.
Subfields: Artificial Intelligence, Biophysics, Computer Vision and Pattern Recognition, Model Architectures
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Quoc V. Le
- Mohit Bansal
- Sergey Levine
- Dacheng Tao
- Jianfeng Gao
- Trevor Darrell
- Percy Liang
In Biophysics
- Richard Zhang
- Jun-Yan Zhu
- Yang Song
- Alexei A. Efros
- Jonathan Ho
- Vladlen Koltun
- Antonio Torralba
- Stefano Ermon
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Xiaogang Wang
- Trevor Darrell
- Jitendra Malik
- Andrew Zisserman
- Yi Yang
In Model Architectures
- Quoc V. Le
- Yoshua Bengio
- Alan L. Yuille
- Jianfeng Gao
- Jiashi Feng
- Vladlen Koltun
- Xiaogang Wang
- Dacheng Tao
Citations in the corpus
- Neural Machine Translation in Linear Time (2016, methodology): “The masked convolutions use dilation to increase the receptive field of the target network (Chen et al., 2014; Yu & Koltun, 2015).”
- Fully convolutional networks for semantic segmentation (2014, background): “esolution by proposals, stacks of learned deconvolution, and unpooling. U-Net [43] combines skip layers and learned deconvolution for pixel labeling of microscopy images. The dilation architecture of [44] makes 3 thorough use of dilated convolution for pixel-precise output without a random field or s”
- Exploring Self-Attention for Image Recognition (2020, background): “These serve as the basis for models developed for other computer vision tasks, such as semantic segmentation [22, 3, 42, 44] and object detection [10, 9, 28, 21].”
- Local Relation Networks for Image Recognition (2019, methodology): “This has been done to enlarge the receptive field, such as through atrous/dilated convolution [4, 33], and to enhance the ability to model geometric deformation, via active [14] and deformable convolution [6, 36].”
- CCNet: Criss-Cross Attention for Semantic Segmentation (2018, methodology): “[37] removed the last two downsample layers to obtain dense prediction and utilized dilated convolutions to enlarge the receptive field.”
- Making Convolutional Networks Shift-Invariant Again (2019, methodology): “Deep networks can also be evaluated densely, by removing striding and making appropriate changes to subsequent layers by using ´a trous /dilated convolutions (Chen et al., 2015; 2018; Yu & Koltun, 2016; Yu et al., 2017).”
- Bag of Tricks for Image Classification with Convolutional Neural Networks (2018, methodology): “[31], we replace the base network with various pre-trained models discussed in previous sections and apply dilation network strategy [2, 29] on stage-3 and stage-4.”
- Unified Perceptual Parsing for Scene Understanding (2018, methodology): “To enable high-resolution predictions for semantic segmentation, dilated convolution [14], a technique which removes the stride of convolutional layers and adds holes between each location of convolution filters, has been proposed to ease the side effect of down-sampling while maintaining the…”
- Semantic Understanding of Scenes Through the ADE20K Dataset (2016, methodology): “ annotation to a wider knowledge base. As for baseline networks for scene parsing on our benchmark, we train several semantic segmentation networks: SegNet [1], FCN-8s [20], DilatedVGG, DilatedResNet [5, 36], two cascade networks proposed in [39] where the backbone models are SegNet and DilatedVGG. ”
- Colorful Image Colorization (2016, methodology): “Yu et al. [22] found that removing downsample striding and accumulating the stride factors as dilations in subsequent layers improved performance.”
- UPSNet: A Unified Panoptic Segmentation Network (2019, background): “As context is crucial for semantic segmentation, one notable improvement to most convolutional models emerged from dilated convolutions [37, 38] which allows for a larger receptive field without the need of more free parameters.”
- Panoptic Feature Pyramid Networks (2019, methodology): “To increase feature resolution, which is necessary for generating highquality results, recent top methods [12, 58, 5, 59] rely heavily on the use of dilated convolution [57] (also known as atrous convolution [10]).”
- Panoptic Segmentation (2018, background): “Stuff classifiers are usually built on fully convolutional nets [30] with dilations [51, 5] while object detectors often use object proposals [15] and are region-based [37, 14].”
- Deep Watershed Transform for Instance Segmentation (2016, methodology): “To preserve spatial resolution, we exploit a high-capacity, multi-scale information aggregation scheme inspired by popular methods in semantic segmentation [9, 30, 16].”
- InstanceCut: From Edges to Instances with MultiCut (2016, methodology): “The output of the perceptron is refined by a context network of Dilation10 [52] architecture.”
- The Cityscapes Dataset for Semantic Urban Scene Understanding (2016, methodology): “(3) When considering IoUcategory, Dilated10 [79] and FCN8s [41] perform particularly well, indicating that these approaches produce comparatively many confusions between the classes within the same category, c.f . the buses in Fig.”
- Generative Modeling by Estimating Gradients of the Data Distribution (2019, methodology): “In the experiments, our model sθ(x, σ) combines the architecture design of U-Net [46] with dilated/atrous convolution [64, 65, 8]—both of which have been proved very successful in semantic segmentation.”
- Video Pixel Networks (2016, background): “One way to increase the receptive field without much effect on the computational complexity is to use dilated convolutions (Chen et al., 2014; Yu and Koltun, 2015), which make the receptive field grow exponentially, as opposed to linearly, in the number of layers.”
- WaveNet: A Generative Model for Raw Audio (2016, methodology): “Dilated convolutions have previously been used in various contexts, e.g. signal processing (Holschneider et al., 1989; Dutilleux, 1989), and image segmentation (Chen et al., 2015; Yu & Koltun, 2016).”