CARAFE: Content-Aware ReAssembly of FEatures

By Kai Chen, Rui Xu, Ziwei Liu, Chen Change Loy, Dahua Lin. 2019

918 citations recorded by the source corpus.

Feature upsampling is a key operation in a number of modern convolutional network architectures, e.g. feature pyramids. Its design is critical for dense prediction tasks such as object detection and semantic/instance segmentation. In this work, we propose Content-Aware ReAssembly of FEatures (CARAFE), a universal, lightweight and highly effective operator to fulfill this goal. CARAFE has several appealing properties: (1) Large field of view. Unlike previous works (e.g. bilinear interpolation) that only exploit subpixel neighborhood, CARAFE can aggregate contextual information within a large receptive field. (2) Content-aware handling. Instead of using a fixed kernel for all samples (e.g. deconvolution), CARAFE enables instance-specific content-aware handling, which generates adaptive kernels on-the-fly. (3) Lightweight and fast to compute. CARAFE introduces little computational overhead and can be readily integrated into modern network architectures. We conduct comprehensive evaluations on standard benchmarks in object detection, instance/semantic segmentation and inpainting. CARAFE shows consistent and substantial gains across all the tasks (1.2% AP, 1.3% AP, 1.8% mIoU, 1.1dB respectively) with negligible computational overhead. It has great potential to serve as a strong building block for future research. Code and models are available at https://github.com/open-mmlab/mmdetection.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

Dynamic Neural Networks: A Survey (2021, background): “I & S Image segmentation [32], [119], [136], [138], [140], [144], [146], [193], [194], [195], [196], [197], Image-to-image translation [198], Object detection [105], [106], [137], [138], [153], Semantic image synthesis [199], [200], [201], Computer Image denoising [202], Fine-grained classification ”
Learning to Upsample by Learning to Sample (2023, methodology): “CARAFE [37] uses a sub-network to generate content-aware dynamic convolution kernels to reassemble the input feature.”
Self-Supervised Scene De-Occlusion (2020, background): “, object detection [4, 5, 6, 7], scene parsing [8, 9, 10], and instance segmentation [11, 12, 13, 14].”
AFPN: Asymptotic Feature Pyramid Network for Object Detection (2023, methodology): “CARAFE [19] is a universal, lightweight, and efﬁcient upsampling operator that can aggregate large receptive ﬁeld information.”
Dynamic Refinement Network for Oriented and Densely Packed Object Detection (2020, methodology): “Similarly, CARAFE [38] proposes a kernel prediction module which is responsible for generating the reassembly kernels in a content-aware manner.”
Involution: Inverting the Inherence of Convolution for Visual Recognition (2021, background): “There also exist previous works that adopt pixel-wise dynamic kernels for feature aggregation, but they mainly capitalize on the context information for feature up-sampling [35, 43] and still rely on convolution for basic feature extraction.”
Delving Deeper into Anti-Aliasing in ConvNets (2020, neutral): “sampling layer for anti-aliasing, whereas the dynamic ﬁltering layer is directly linked to the prediction (last) layer in order to incorporate motion information for video recognition tasks. Finally, [33] introduces an adaptive convolution layer for upsampling, whereas we focus on downsampling with ”
NAS-DIP: Learning Deep Image Prior with Neural Architecture Search (2020, background): “, CARAFE [76]) can also be incorporated in our search space easily in the future.”
Seesaw Loss for Long-Tailed Instance Segmentation (2020, methodology): “The enhanced FPG uses deformable convolution v2 (DCNv2) [59] after feature upsampling, and a downsampler version of CARAFE [47, 48] for feature downsampling.”
Towards Large-Scale Small Object Detection: Survey and Benchmarks (2022, background): “This situation will likely get worse for those objects with extremely limited sizes [108].”
Face Anti-Spoofing with Human Material Perception (2020, background): “Unlike [52] which aims for feature upsampling, here we focus on the general multi-level feature reﬁnement.”
Positional Encoding as Spatial Inductive Bias in GANs (2020, background): “It is widely adopted for differentiable image warping [38, 44, 49, 51, 50, 48] and aligning pixels among different spaces [10, 17, 27, 40, 41, 53].”