Pattern-Affinitive Propagation Across Depth, Surface Normal and Semantic Segmentation

By Zhenyu Zhang, Zhen Cui, Chunyan Xu, Yan Yan, Nicu Sebe, Jian Yang. 2019

322 citations recorded by the source corpus.

In this paper, we propose a novel Pattern-Affinitive Propagation (PAP) framework to jointly predict depth, surface normal and semantic segmentation. The motivation behind it comes from the statistic observation that pattern-affinitive pairs recur much frequently across different tasks as well as within a task. Thus, we can conduct two types of propagations, cross-task propagation and task-specific propagation, to adaptively diffuse those similar patterns. The former integrates cross-task affinity patterns to adapt to each task therein through the calculation on non-local relationships. Next the latter performs an iterative diffusion in the feature space so that the cross-task affinity patterns can be widely-spread within the task. Accordingly, the learning of each task can be regularized and boosted by the complementary task-level affinities. Extensive experiments demonstrate the effectiveness and the superiority of our method on the joint three tasks. Meanwhile, we achieve the state-of-the-art or competitive results on the three related datasets, NYUD-v2, SUN-RGBD and KITTI.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Reparameterizing Convolutions for Incremental Multi-Task Learning without Task Interference (2020, methodology): “Decoder-focused approaches start from the feature representations learned at the encoding stage, and further refine them at the decoding stage by distilling information across tasks in a one-off [63], sequential [65], recursive [66], or even multi-scale [62] manner.”
Geometry Uncertainty Projection Network for Monocular 3D Object Detection (2021, background): “Many works focus on task relation representation learning [28, 45, 48, 51, 52, 53].”
CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation With Transformers (2022, methodology): “Moreover, there are multi-task frameworks [48], [49] that facilitate inter-task feature propagation for RGB-D scene understanding, but they rely on supervision from other tasks for joint learning.”
Self-Supervised Monocular Depth Estimation: Solving the Dynamic Object Problem by Semantic Guidance (2020, background): “, domain adaptation [3,41,69], depth estimation [11,47,68] and semantic segmentation [28,30].”
Bi-directional Cross-Modality Feature Propagation with Separation-and-Aggregation Gate for RGB-D Semantic Segmentation (2020, methodology): “2 PAP [44] 50.”
GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Image (2024, background): “On the other hand, several works [12,28,67,79] create multiple branches for depth and normal, and enforce information exchange through propagating latent features.”
Spatial Information Guided Convolution for Real-Time RGBD Semantic Segmentation (2020, methodology): “learning [7], [46]–[50] or spatial-temporal analysis [51], are further used to improve segmentation accuracy.”
MTI-Net: Multi-Scale Task Interaction Networks for Multi-Task Learning (2020, methodology): “In [51], they extended upon this idea.”
Multi-Task Learning with Deep Neural Networks: A Survey (2020, background): “Pattern-Affinitive Propagation (PAP) (Zhang et al., 2019) expands on this architecture by introducing an affinity learning layer which learns to represent pair-wise relationships of tasks and combines features from various tasks according to these relationships.”
Neural Window Fully-connected CRFs for Monocular Depth Estimation (2022, background): “Also, other approaches bring in auxiliary information to help the training of the depth network, such as sparse depth [10] or segmentation information [15,24,27,44].”