TensorMask: A Foundation for Dense Object Segmentation

By Xinlei Chen, Ross B. Girshick, Kaiming He, Piotr Doll'ar. 2019

360 citations recorded by the source corpus.

Sliding-window object detectors that generate bounding-box object predictions over a dense, regular grid have advanced rapidly and proven popular. In contrast, modern instance segmentation approaches are dominated by methods that first detect object bounding boxes, and then crop and segment these regions, as popularized by Mask R-CNN. In this work, we investigate the paradigm of dense sliding-window instance segmentation, which is surprisingly under-explored. Our core observation is that this task is fundamentally different than other dense prediction tasks such as semantic segmentation or bounding-box object detection, as the output at every spatial location is itself a geometric structure with its own spatial dimensions. To formalize this, we treat dense instance segmentation as a prediction task over 4D tensors and present a general framework called TensorMask that explicitly captures this geometry and enables novel operators on 4D tensors. We demonstrate that the tensor view leads to large gains over baselines that ignore this structure, and leads to results comparable to Mask R-CNN. These promising results suggest that TensorMask can serve as a foundation for novel advances in dense mask prediction and a more complete understanding of the task. Code will be made available.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

K-Net: Towards Unified Image Segmentation (2021, methodology): “Recent attempts [10, 55, 56, 60] perform instance segmentation in one stage without involving detection nor embedding generation.”
Class-Incremental Instance Segmentation via Multi-Teacher Networks (2021, methodology): “Two-stage methods formulate this task as the paradigm of “Detect then Segment” (Li et al. 2017; He et al. 2017; Li-u et al. 2018; Chen et al. 2019).”
PointRend: Image Segmentation As Rendering (2019, methodology): “TensorMask [6], an alternative sliding-window method, uses a sophisticated network design to predict sharp high-resolution masks for large objects, but its accuracy also lags slightly behind.”
Image Segmentation Using Deep Learning: A Survey (2020, background): “[67], is based on dense sliding window instance segmentation.”
PolarMask: Single Shot Instance Segmentation With Polar Representation (2019, methodology): “3 TensorMask [4] ResNet-101-FPN 72 37.”
Evolution of Image Segmentation using Deep Convolutional Neural Network: A Survey (2020, methodology): “ment step and SSD[23], RetinaNet[26] without using renement stage. Though sliding window approach is popular in object detection but it was missing in case of instance segmentation task. Chen et al. [158] have introduced dense instance segmentation to ll this gap and introduced TensorMask. Recently,”
SOLOv2: Dynamic and Fast Instance Segmentation (2020, methodology): “Top-down methods [20, 12, 25, 14, 6, 2, 4, 38] solve the problem from the perspective of object detection, i . e ., detecting ﬁrst and then segmenting the object in the box.”
Crossover Learning for Fast Online Video Instance Segmentation (2021, background): “Many prior works [24, 10, 14, 27, 9, 12, 5, 58, 52, 30, 6, 31] contribute a lot to the rapid developments in this field.”
Interactive Object Segmentation With Inside-Outside Guidance (2022, neutral): “…a revolutionary advancement in semantic [9], [10], [11], [12], [13], [14], [15], [16], [17], [18] and instance segmentation [19], [20], [21], [22], [23], [24], [25], [26], [27], [28], [29] for different domains, such as general scenes [3], [30], [31], autonomous driving [5], [32], [33], aerial…”
Transformer-Based Visual Segmentation: A Survey (2023, methodology): “Besides, there are also several approaches [85], [86] using gird representation to learn instance masks directly.”
SipMask: Spatial Information Preservation for Fast Image and Video Instance Segmentation (2020, methodology): “For a fair comparison with the state-of-the-art single-stage methods [11,2], we follow the 6×, multi-scale training scheme.”
STEm-Seg: Spatio-temporal Embeddings for Instance Segmentation in Videos (2020, methodology): “Image-level Instance Segmentation: Several existing methods for imagelevel instance segmentation, which is closely related to our task, operate in a top-down fashion by using a Region Proposal Network to generate object proposals [13,71,72,75], which are then classified and segmented [32].”
CenterMask: Real-Time Anchor-Free Instance Segmentation (2019, methodology): “In less than half training epochs, CenterMask also surpasses the dense sliding window method, TensorMask [5], by 1.”
A survey on instance segmentation: state of the art (2020, methodology): “TensorMask [91] uses a different architecture as compared to the techniques discussed in Sect.”
MPDIoU: A Loss for Efficient and Accurate Bounding Box Regression (2023, background): “A typical form of L is `n-norm, for example, mean-square error (MSE) loss and Smooth-`1 loss [43], which have been widely adopted in object detection [44]; pedestrian detection [45, 46]; scene text spotting [34, 47]; 3D object detection [48, 49]; pose estimation [50, 51]; and instance segmentation [”
Vision-based robotic grasping from object localization, object pose estimation to grasp estimation for parallel grippers: a review (2020, background): “, 2017], TensorMask [Chen et al., 2019c], YOLACT [Bolya et al.”