Learning Affinity via Spatial Propagation Networks

By Sifei Liu, Shalini De Mello, Jinwei Gu, Guangyu Zhong, Ming-Hsuan Yang, Jan Kautz. 2017

318 citations recorded by the source corpus.

In this paper, we propose spatial propagation networks for learning the affinity matrix for vision tasks. We show that by constructing a row/column linear propagation model, the spatially varying transformation matrix exactly constitutes an affinity matrix that models dense, global pairwise relationships of an image. Specifically, we develop a three-way connection for the linear propagation model, which (a) formulates a sparse transformation matrix, where all elements can be the output from a deep CNN, but (b) results in a dense affinity matrix that effectively models any task-specific pairwise similarity matrix. Instead of designing the similarity kernels according to image features of two points, we can directly output all the similarities in a purely data-driven manner. The spatial propagation network is a generic framework that can be applied to many affinity-related tasks, including but not limited to image matting, segmentation and colorization, to name a few. Essentially, the model can learn semantically-aware affinity values for high-level vision tasks due to the powerful learning capability of the deep neural network classifier. We validate the framework on the task of refinement for image segmentation boundaries. Experiments on the HELEN face parsing and PASCAL VOC-2012 semantic segmentation tasks show that the spatial propagation network provides a general, effective and efficient solution for generating high-quality segmentation results.

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

CCNet: Criss-Cross Attention for Semantic Segmentation (2018, neutral): “[24] utilized RNNs to capture long-range dependencies.”
Non-local Neural Networks (2017, background): “The iterative mean-field inference of CRF can be turned into a recurrent network and trained [56, 42, 8, 18, 34].”
Representation Compensation Networks for Continual Semantic Segmentation (2022, methodology): “Some methods [14,22,32,37,38,50,55] is inspired by Non-local [85], utilizing attention mechanisms to establish connections between image contexts.”
Strip Pooling: Rethinking Spatial Pooling for Scene Parsing (2020, background): “Besides, aggregating non-local context [36, 58, 29, 15, 7, 45, 21, 53, 66, 23, 14] is also effective for scene parsing.”
SSAP: Single-Shot Instance Segmentation With Affinity Pyramid (2019, methodology): “The concept of learning pixel-pair affinity has been developed in many previous works [36, 23, 1, 4, 42] to facilitate semantic segmentations during training or postprocessing.”
Pattern-Affinitive Propagation Across Depth, Surface Normal and Semantic Segmentation (2019, methodology): “As M̂ is normalized and has finite Frobenius norm, the stability of such PDE can be guaranteed [39].”
Pixel-Adaptive Convolutional Neural Networks (2019, background): “[30] propose differentiable local propagation modules that have roots in domain transform ﬁl-tering [14].”
Learning Depth with Convolutional Spatial Propagation Network (2018, methodology): “Here, following SPN [25], we normalize kernel weights to the range of ð 1; 1Þ so that the model can be stabilized when the condition P a;b;a;b 61⁄40 jki;jða; bÞj 1 is satisfied.”
Depth Estimation via Affinity Learned with Convolutional Spatial Propagation Network (2018, methodology): “[14], we branch an additional output from the given network, which shares the same feature extractor with the depth network.”
GA-Net: Guided Aggregation Net for End-To-End Stereo Matching (2019, methodology): “The same aggregation steps were used in MC-CNN [23, 30], and similar iterative steps were employed in [1, 2, 14].”
Object Instance Annotation With Deep Extreme Level Set Evolution (2019, methodology): “[27] employs deep CNNs to learn an affinity matrix and refines the segmentation output via spatial propagation.”
A Survey on Deep Learning Techniques for Stereo-Based Depth Estimation (2020, background): “Convolutional Spatial Propagation Networks (CSPN) [98], [99], which implement an anisotropic diffusion process, are particularly suitable for depth completion since they predict the diffusion tensor using a deep CNN.”
SegFix: Model-Agnostic Boundary Refinement for Segmentation (2020, result): “In general, we only need to train a single unified SegFix model to improve the boundary quality of various segmentation models across different datasets, thus SegFix is much more training friendly (and saves a lot of energy consumption) compared to the previous methods [5,52,17,41,40,30] that requir”
Understanding Deep Learning Techniques for Image Segmentation (2019, background): “Similarly, in the works of [126], a row/columnwise propagation model was proposed the calculated the global pairwise relationship across an image.”
Adaptive Affinity Field for Semantic Segmentation (2018, background): “Besides, low-level cues, such as affinity [25,26] and contour [27,28] have also been leveraged to encode image structures.”
Videos as Space-Time Region Graphs (2018, methodology): “Instead of using mean-field inference, variant simpler feedforward graph based neural network have been proposed recently [67,68,19,69,70,71].”
PENet: Towards Precise and Efficient Image Guided Depth Completion (2021, background): “The spatial propagation network (SPN) is proposed by Liu et al. [25] to learn local afﬁnities that can be exploited in various high-level vision tasks.”
Improving Semantic Segmentation via Decoupled Body and Edge Supervision (2020, methodology): “DCN [41] uses predicted oﬀset to aggregate information in kernel and SPN [42] proposes to propagate information through aﬃnity pairs.”