Pixel-Adaptive Convolutional Neural Networks

By Hang Su, Varun Jampani, Deqing Sun, Orazio Gallo, Eric Learned-Miller, Jan Kautz. 2019

337 citations recorded by the source corpus.

Convolutions are the fundamental building blocks of CNNs. The fact that their weights are spatially shared is one of the main reasons for their widespread use, but it is also a major limitation, as it makes convolutions content-agnostic. We propose a pixel-adaptive convolution (PAC) operation, a simple yet effective modification of standard convolutions, in which the filter weights are multiplied with a spatially varying kernel that depends on learnable, local pixel features. PAC is a generalization of several popular filtering techniques and thus can be used for a wide range of use cases. Specifically, we demonstrate state-of-the-art performance when PAC is used for deep joint image upsampling. PAC also offers an effective alternative to fully-connected CRF (Full-CRF), called PAC-CRF, which performs competitively compared to Full-CRF, while being considerably faster. In addition, we also demonstrate that PAC can be used as a drop-in replacement for convolution layers in pre-trained networks, resulting in consistent performance improvements.

Subfields: Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Visual Transformers: Token-based Image Representation and Processing for Computer Vision (2020, background): “Later work [46, 33, 39] interpret this “modulation” as a way to make convolution spatially adaptive and content-aware.”
Dynamic Neural Networks: A Survey (2021, background): “Unlike [107] that requires a sub-network for feature embedding, pixel-adaptive convolution (PAC) [108] adapts the convolutional weights based on the attention mask generated from the input feature at each layer.”
Single-Stage Semantic Segmentation From Image Labels (2020, methodology): “Instead, our implementation derives from the PixelAdaptive Convolution (PAC) [49].”
Spatially-Attentive Patch-Hierarchical Network for Adaptive Motion Deblurring (2020, methodology): “Our work is based on [19] as we use a meta-layer to generate pixel dependent spatially varying kernel to implement spatially variant convolution operation.”
SqueezeSegV3: Spatially-Adaptive Convolution for Efficient Point-Cloud Segmentation (2020, methodology): “In our work, we propose a more general framework such that channel-wise attention [14,13], spatial attention [51,58] and PAC [42] can be considered as special cases of spatially-adaptive convolution.”
Involution: Inverting the Inherence of Convolution for Visual Recognition (2021, background): “There also exist previous works that adopt pixel-wise dynamic kernels for feature aggregation, but they mainly capitalize on the context information for feature up-sampling [35, 43] and still rely on convolution for basic feature extraction.”
Focal Sparse Convolutional Networks for 3D Object Detection (2022, background): “Methods of [39, 42, 45, 50] seek spatial-wise sparsity for efficient inference.”
SOLOv2: Dynamic and Fast Instance Segmentation (2020, background): “Pixel-adaptive convolution [29] multiplies the weights of the ﬁlters and a spatially varying kernel to make the standard convolution content-adaptive.”
Few-shot Video-to-Video Synthesis (2019, background): “This class of networks has a different inductive bias to regular networks and has found use in several tasks including sequence modeling [15], image ﬁltering [23, 59, 49], frame interpolation [38, 37], and neural architecture search [64].”
Learning to Predict Layout-to-image Conditional Convolutions for Semantic Image Synthesis (2019, positive): “Su et al. [28] proposed pixel-adaptive CNN which multiplies the conventional convolutional ﬁlter with a spatially-varying kernel to obtain convolutional kernels.”
Attention-Based Transformers for Instance Segmentation of Cells in Microstructures (2020, neutral): “[31] 5 .”
HiFaceGAN: Face Renovation via Collaborative Suppression and Replenishment (2020, background): “Therefore, it is highly desirable to select and aggregate informative features with content-adaptive filters, such as LIP [10] or PAC [51].”
A Survey of Visual Transformers (2021, methodology): “It inherits the cross-attention of original mask head in DETR and substitutes the addition operation with a pixel-adaptive convolution [130] for feature fusion.”
Delving Deeper into Anti-Aliasing in ConvNets (2020, background): “ever, directly integrating these modules into a neural network requires careful tuning of hyperparameters subject to the input image (e.g., s s and s r in bilateral ﬁlter or r and e in guided ﬁlter). [16,29] introduced the dynamic ﬁltering layer, whose weights are predicted by convolution layers con”