Drop an Octave: Reducing Spatial Redundancy in Convolutional Neural Networks With Octave Convolution

By Yunpeng Chen, Haoqi Fan, Bing Xu, Zhicheng Yan, Yannis Kalantidis, Marcus Rohrbach, Shuicheng Yan, Jiashi Feng. IEEE International Conference on Computer Vision, 2019

687 citations recorded by the source corpus.

In natural images, information is conveyed at different frequencies where higher frequencies are usually encoded with fine details and lower frequencies are usually encoded with global structures. Similarly, the output feature maps of a convolution layer can also be seen as a mixture of information at different frequencies. In this work, we propose to factorize the mixed feature maps by their frequencies, and design a novel Octave Convolution (OctConv) operation to store and process feature maps that vary spatially “slower” at a lower spatial resolution reducing both memory and computation cost. Unlike existing multi-scale methods, OctConv is formulated as a single, generic, plug-and-play convolutional unit that can be used as a direct replacement of (vanilla) convolutions without any adjustments in the network architecture. It is also orthogonal and complementary to methods that suggest better topologies or reduce channel-wise redundancy like group or depth-wise convolutions. We experimentally show that by simply replacing convolutions with OctConv, we can consistently boost accuracy for both image and video recognition tasks, while reducing memory and computational cost. An OctConv-equipped ResNet-152 can achieve 82.9% top-1 classification accuracy on ImageNet with merely 22.2 GFLOPs.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/0d9eb72ea89bb7bf5720a7b2c2f3f77c26fc67a6

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Inception Transformer (2022, background): “CNNs [43–47] are the de-facto model for vision tasks due to their outstanding ability to model local dependency [47–49] as well as extract high-frequency [19, 50].”
CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification (2021, methodology): “In the context of CNNs, multi-scale feature representations have been used for detection and recognition of objects at multiple scales [4, 22, 44, 26], as well as to speed up neural networks in Big-Little Net [5] and OctNet [6].”
Glance and Focus: a Dynamic Approach to Reducing Spatial Redundancy in Image Classification (2020, positive): “The OctConv [6] reduces the spatial resolution by using low-frequency features.”
Gate Decorator: Global Filter Pruning Method for Accelerating Deep Convolutional Neural Networks (2019, background): “[2] designed a plug-and-play convolutional unit named OctConv, which factorizes the mixed feature maps by their frequencies.”
SCConv: Spatial and Channel Reconstruction Convolution for Feature Redundancy (2023, methodology): “The related SOTA approaches include OctConv [3], Ghost-Net [8], SPConv [32], SlimConv [23], TiedConv [28].”
Fast Vision Transformers with HiLo Attention (2022, methodology): “Furthermore, Octave convolution [9] targeted convolutional layers and proposed to locally applies convolution on high/low-resolution feature maps, separately.”
AdversarialNAS: Adversarial Neural Architecture Search for GANs (2019, background): “On the other hand, the benefits of specially designing the network architecture have been proven through lost of discriminative networks, such as ResNet [17], DenseNet [20], MobileNet [37], ShuffleNet [49], EfficientNet [39], HRNet [38] and [6, 10].”
Run, Don't Walk: Chasing Higher FLOPS for Faster Neural Networks (2023, background): “Though there have been numerous studies [7, 8, 21, 33, 55, 56, 83, 86] to achieve higher efficiency, the rationale behind them is more or less to perform a low-rank approximation.”
Multiscale Vision Transformers (2021, background): “Incorporating downsampling, shift invariance, and shared weights, ConvNets are de-facto standard backbones for computer vision tasks for image [70, 67, 94, 96, 51, 15, 18, 39, 99, 87, 46] and video [93, 36, 14, 85, 74, 112, 102, 34, 111, 40, 33, 123, 62].”
Wavelet Convolutions for Large Receptive Fields (2024, methodology): “However, the methods from [1,15,33] are highly-customized for specific architectures and cannot seamlessly be used within other CNN architectures, while [3] focuses on computational efficiency.”
Res2Net: A New Multi-Scale Backbone Architecture (2019, background): “One common operation in [5], [9], [11], [48], [49] is that they all use pooling or up-sample to re-size the feature map to 2 times of the original scale to save the computational budget while maintaining or even improving performance.”
Diverse Branch Block: Building a Convolution as an Inception-like Unit (2021, methodology): “For examples, Squeeze-and-Excitation (SE) block [14] and Efﬁcient Convolutional Attention (ECA) block [29] utilizes the attention mechanism to recalibrate the features, Octave Convolution [3] reduces the spatial redundancy of regular convolution, Deformable Convolution [6] augments the spatial sampl”
X3D: Expanding Architectures for Efficient Video Recognition (2020, methodology): “Finally, in the lower compute regime X3D-M is comparable to SlowFast 4×16, R50 and Oct-I3D + NL [8] while having 4.”
DO-Conv: Depthwise Over-Parameterized Convolutional Layer (2020, background): “Several alternatives to the classical convolutional layer have been proposed, which o ﬀ er improved feature learning capability and / or e ﬃ ciency [9, 35, 24, 6].”
MViTv2: Improved Multiscale Vision Transformers for Classification and Detection (2021, background): “CNNs serve as the primary backbones for computer vision tasks, including image recognition [10,15,34,39,46,48,62, 67,69,71], object detection [6,29,38,53,63,93] and video recognition [8,22,23,25,28,42,51,61,66,75,79,84,92].”