Segmentation-Aware Convolutional Networks Using Local Attention Masks

By Adam W. Harley, Konstantinos G. Derpanis, Iasonas Kokkinos. 2017

152 citations recorded by the source corpus.

We introduce an approach to integrate segmentation information within a convolutional neural network (CNN). This counter-acts the tendency of CNNs to smooth information across regions and increases their spatial precision. To obtain segmentation information, we set up a CNN to provide an embedding space where region co-membership can be estimated based on Euclidean distance. We use these embeddings to compute a local attention mask relative to every neuron position. We incorporate such masks in CNNs and replace the convolution operation with a “segmentation-aware” variant that allows a neuron to selectively attend to inputs coming from its own region. We call the resulting network a segmentation-aware CNN because it adapts its filters at each image point according to local segmentation cues, while at the same time remaining fully-convolutional. We demonstrate the merit of our method on two widely different dense prediction tasks, that involve classification (semantic segmentation) and regression (optical flow). Our results show that in semantic segmentation we can replace DenseCRF inference with a cascade of segmentation-aware filters, and in optical flow we obtain clearly sharper responses than the ones obtained with comparable networks that do not use segmentation. In both cases segmentation-aware convolution yields systematic improvements over strong baselines.

Subfields: Biophysics, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biophysics

In Computer Vision and Pattern Recognition

Citations in the corpus

Non-local Neural Networks (2017, background): “The iterative mean-field inference of CRF can be turned into a recurrent network and trained [56, 42, 8, 18, 34].”
Dynamic Neural Networks: A Survey (2021, methodology): “For example, segmentation-aware convolutional network [108] applies locally masked convolution to aggregate information with larger weights from”
Image Inpainting for Irregular Holes Using Partial Convolutions (2018, methodology): “The partial convolution at every location, similarly defined in [7], is expressed as:”
Recurrent Pixel Embedding for Instance Grouping (2017, methodology): “This is in spirit similar to the model in [33], which considers local consistency to improve spatial precision.”
Transparency by Design: Closing the Gap Between Performance and Interpretability in Visual Reasoning (2018, background): “Several authors [7, 10, 25] have proposed attention mechanisms that a network can use, optionally.”
Pixel-Adaptive Convolutional Neural Networks (2019, background): “, motion prediction [47], semantic segmentation [17], and Monte Carlo rendering denoising [4].”
Iterative Residual Refinement for Joint Optical Flow and Occlusion Estimation (2019, background): “To address this, bilateral filters or local attention maps [21, 24] have been proposed as viable solutions.”
Exploring Cross-Image Pixel Contrast for Semantic Segmentation (2021, methodology): “Although a few prior methods address the idea of metric learning in semantic segmentation, they only account for the local content from objects [29] or instances [16, 1, 22, 42].”
Rethinking Semantic Segmentation: A Prototype View (2022, background): “Since it was proposed, numerous efforts have been devoted to improving FCN, by, for example, enlarging the receptive field [15, 16, 25, 124, 128, 135]; strengthening context cues [4, 43, 47, 48, 56, 57, 70, 75, 77, 81, 90, 126, 128, 129, 132, 138, 141]; leveraging boundary information [6, 14, 27, 66”
Semi-convolutional Operators for Instance Segmentation (2018, methodology): “It has been used for body joints [41], semantic segmentation [1,21,6] and optical flow [1], and, more relevant to our work, to instance segmentation [15,12,6,28].”
Videos as Space-Time Region Graphs (2018, methodology): “In the context of deep learning, especially for semantic segmentation, the CRF model is often applied on the outputs of the ConvNets by performing mean-field inference [61,62,63,64,65,66].”
YOLACT++ Better Real-Time Instance Segmentation (2019, methodology): “Finally, some methods first perform semantic segmentation followed by boundary detection [19], pixel clustering [20], [21], CRF inference [22], or learn an embedding to form instance masks [23], [24], [25], [26].”
Dynamic Filtering with Large Sampling Field for ConvNets (2018, methodology): “Even though SegAware attention model [10] explicitly takes advantage of boundary information which requires additional training data, our LS-DFN can still slightly outperforms them using FlowNetS as baseline model.”
Partial Convolution based Padding (2018, background): “Harley et al. [6] uses soft attention masks to reweight the convolution results for semantic segmentation.”