Learning to Fuse Things and Stuff

By Jie Li, Allan Raventos, Arjun Bhargava, Takaaki Tagawa, Adrien Gaidon. 2018

104 citations recorded by the source corpus.

We propose an end-to-end learning approach for panoptic segmentation, a novel task unifying instance (things) and semantic (stuff) segmentation. Our model, TASCNet, uses feature maps from a shared backbone network to predict in a single feed-forward pass both things and stuff segmentations. We explicitly constrain these two output distributions through a global things and stuff binary mask to enforce cross-task consistency. Our proposed unified network is competitive with the state of the art on several benchmarks for panoptic segmentation as well as on the individual semantic and instance segmentation tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation (2020, background): “TASCNet [48] and AUNet [51] propose a module to guide the fusion between ‘thing’ and ‘stuff’ predictions, while Liu et al .”
UPSNet: A Unified Panoptic Segmentation Network (2019, methodology): “We compare our model on Cityscapes with Li et al. [19], the combined method [17] and TASCNet [18].”
Panoptic Feature Pyramid Networks (2019, methodology): “TRI-ML used separate networks for the challenge but a joint network in their recent updated tech report [33] (which cites a preliminary version of our work).”
Panoptic Segmentation (2018, neutral): “[23, 47, 48, 27, 22, 21, 17] address PS).”
MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers (2020, methodology): “Then, the instance segments (‘thing’) and semantic segments (‘stuff’) [13] are fused by merging modules [54, 56, 74, 67, 101] to generate panoptic segmentation.”
Panoptic-DeepLab: A Simple, Strong, and Fast Baseline for Bottom-Up Panoptic Segmentation (2019, methodology): “During evaluation, due to the sensitivity of PQ [15, 8, 12], we re-assign to ‘VOID’ label all ‘stuff’ segments whose areas are smaller than a threshold.”
Unifying Training and Inference for Panoptic Segmentation (2020, methodology): “A number of works have also attempted to encourage consistency between semantic and instance predictions by adding a communication mechanism between the two subnets [16, 18].”
AdaptIS: Adaptive Instance Selection Network (2019, methodology): “5 - - - TASCNet [17] ImageNet+COCO ResNet-50 59.”
Auto-Panoptic: Cooperative Multi-Component Architecture Search for Panoptic Segmentation (2020, unknown): “Previous methods [28, 40, 23, 22] use complicated fusion modules to remap the outputs from two branches to a canvas which is of the same size as the semantic segmentation feature map to align foreground masks with background stuff, and thus introduces a considerable amount of parameters and high…”
Scaling Wide Residual Networks for Panoptic Segmentation (2020, methodology): “Top-down approaches [43, 67, 50, 54, 60, 94, 51, 10, 47, 91] typically pair Mask R-CNN [27] with a light-weight ‘stuff‘ segmentation branch, while bottom-up approaches [95, 20, 87, 12, 88] group ‘thing‘ pixels from semantic segmentation predictions.”
Seamless Scene Segmentation (2019, methodology): “1 % PQ score over DeeperLab [55] and TASCNet [29], respectively, despite using a weaker backbone compared to DeeperLab and not pre-training on COCO as opposed to TASCNet.”
SpatialFlow: Bridging All Tasks for Panoptic Segmentation (2019, background): “While recent works [6], [7], [8], [10], [11] for panoptic segmentation focus on how to combine the outputs of segmentation tasks properly but ignore to highlight the significance of the object location in the training phase.”
Bidirectional Graph Reasoning Network for Panoptic Segmentation (2020, methodology): “The panoptic segmentation task is to assign each pixel in an image a semantic label and an instance id. Current methods typically address this issue with a uniﬁed model using two branches for foreground things and background stuff separately [8, 19, 21, 22].”
Naive-Student: Leveraging Semi-Supervised Learning in Video Sequences for Urban Scene Segmentation (2020, result): “Similar to other works on panoptic segmentation [38,43,37,85,61,87,78], we re-assign to void label all ‘stuff‘ segments whose areas are smaller than a threshold of 4096.”
Fully Convolutional Networks for Panoptic Segmentation (2020, unknown): “TASCNet [21] Res50-FPN 32.”
SOGNet: Scene Overlap Graph Network for Panoptic Segmentation (2019, methodology): “…the results with state-of-the-art methods including the method in (Li, Arnab, and Torr 2018), JSIS (de Geus, Meletis, and Dubbelman 2018), TASCNet (Li et al. 2018), Panoptic FPN (Kirillov et al. 2019a), OANet (Liu et al. 2019), AUNet (Li et al. 2019b), UPSNet (Xiong et al. 2019), and OCFusion…”