Hybrid Task Cascade for Instance Segmentation

By Kai Chen, Jiangmiao Pang, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jianping Shi, Wanli Ouyang, Chen Change Loy and 1 more. 2019

1,537 citations recorded by the source corpus.

Cascade is a classic yet powerful architecture that has boosted performance on various tasks. However, how to introduce cascade to instance segmentation remains an open question. A simple combination of Cascade R-CNN and Mask R-CNN only brings limited gain. In exploring a more effective approach, we find that the key to a successful instance segmentation cascade is to fully leverage the reciprocal relationship between detection and segmentation. In this work, we propose a new framework, Hybrid Task Cascade (HTC), which differs in two important aspects: (1) instead of performing cascaded refinement on these two tasks separately, it interweaves them for a joint multi-stage processing; (2) it adopts a fully convolutional branch to provide spatial context, which can help distinguishing hard foreground from cluttered background. Overall, this framework can learn more discriminative features progressively while integrating complementary features together in each stage. Without bells and whistles, a single HTC obtains 38.4% and 1.5% improvement over a strong Cascade Mask R-CNN baseline on MSCOCO dataset. Moreover, our overall system achieves 48.6 mask AP on the test-challenge split, ranking 1st in the COCO 2018 Challenge Object Detection Task. Code is available at https://github.com/open-mmlab/mmdetection.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Benchmarking Detection Transfer Learning with Vision Transformers (2021, background): “Mask R-CNN is the foundation of higher complexity/higher performing systems, such as Cascade R-CNN [4] and HTC/HTC++ [6, 29], which may improve upon the results presented here at the cost of additional complexity that is orthogonal to the goal of benchmarking transfer learning.”
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows (2021, methodology): “On a higher baseline of 52.3 box AP and 46.0 mask AP using an improved HTC framework, the gains by Swin Transformer are also high, at +4.1 box AP and +3.1 mask AP (see Table 2(c)).”
A Simple Framework for Open-Vocabulary Segmentation and Detection (2023, methodology): “For example, HTC [2] and Mask DINO [29] can both learn from large-scale detection data and then be fine-tuned to a specific segmentation dataset.”
Grounded Language-Image Pre-training (2021, methodology): “Though our current implementation is built upon Dynamic Head (DyHead) [10], our unified formulation can be generalized to any object detection systems [4, 6, 9, 10, 10, 33, 45, 46, 69] for scalable Grounded Language-Image Pre-training.”
OneFormer: One Transformer to Rule Universal Image Segmentation (2022, methodology): “Traditional instance segmentation methods [2, 5, 23] are also formulated as mask classifiers, which predict binary masks and a class label for each mask.”
Exploring Plain Vision Transformer Backbones for Object Detection (2022, methodology): “43.1 AP mask ), which uses HTC with CBNetV2 [36] that combines two Swin-L backbones.”
3D Instance Segmentation of MVS Buildings (2021, methodology): “HTC combines detection and segmentation with a multi-task and multi-stage hybrid cascade structure [21].”
Masked-attention Mask Transformer for Universal Image Segmentation (2021, methodology): “performance of universal architectures [14, 62], is currently lower (> 9 AP) than the SOTA specialized architecture for instance segmentation [6].”
K-Net: Towards Unified Image Segmentation (2021, background): “Extensions of this framework add a mask scoring branch [24] or adopt a cascade structure [3, 5].”
Occluded Video Instance Segmentation: A Benchmark (2021, methodology): “By using a larger backbone (STSN [3]-ResNeXt-10164x4d [45]), a better detection network (HybridTask Cascade Network [5]), higher resolution inputs for segmentation network, and more training iterations, it reports a much higher AP of 46.”
Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming (2019, methodology): “We additionally provide results for Faster R-CNN and Mask R-CNN models with deformable convolutions [Dai et al., 2017, Zhu et al., 2018] in Appendix D. Models were evaluated using the mmdetection toolbox [Chen et al., 2019b]; all models were trained and tested with standard hyperparameters.”
MMDetection: Open MMLab Detection Toolbox and Benchmark (2019, methodology): “We benchmark different methods on COCO 2017 val , including SSD [19], RetinaNet [18], Faster RCNN [27], Mask RCNN [13] and Cascade R-CNN [18], Hybrid Task Cascade [4] and FCOS [32].”
Deep High-Resolution Representation Learning for Visual Recognition (2019, methodology): “We also evaluate the performance of joint detection and instance segmentation, under three frameworks: Mask R-CNN [53], Cascade Mask R-CNN [13], and Hybrid Task Cascade [16].”
ResNeSt: Split-Attention Networks (2020, neutral): “0 HTC [4] ResNet50 43.”
PointRend: Image Segmentation As Rendering (2019, methodology): “Instance segmentation methods based on the Mask R-CNN meta-architecture [17] occupy top ranks in recent challenges [29, 2].”
Swin Transformer V2: Scaling Up Capacity and Resolution (2021, methodology): “Then we ﬁne-tune the HTC++ model on COCO dataset, with the mask branch randomly initialized and other model weights loaded from the Objects-365-V2 pre-trained model.”