Shift: A Zero FLOP, Zero Parameter Alternative to Spatial Convolutions

By Bichen Wu, Alvin Wan, Xiangyu Yue, Peter H. Jin, Sicheng Zhao, Noah Golmant, Amir Gholaminejad, Joseph E. Gonzalez, Kurt Keutzer. 2017

401 citations recorded by the source corpus.

Neural networks rely on convolutions to aggregate spatial information. However, spatial convolutions are expensive in terms of model size and computation, both of which grow quadratically with respect to kernel size. In this paper, we present a parameter-free, FLOP-free "shift" operation as an alternative to spatial convolutions. We fuse shifts and point-wise convolutions to construct end-to-end trainable shift-based modules, with a hyperparameter characterizing the tradeoff between accuracy and efficiency. To demonstrate the operation's efficacy, we replace ResNet's 3x3 convolutions with shift-based modules for improved CIFAR10 and CIFAR100 accuracy using 60% fewer parameters; we additionally demonstrate the operation's resilience to parameter reduction on ImageNet, outperforming ResNet family members. We finally show the shift operation's applicability across domains, achieving strong performance with fewer parameters on image classification, face verification and style transfer.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Efficiency & Compression

Citations in the corpus

Visual Transformers: Token-based Image Representation and Processing for Computer Vision (2020, neutral): “Early work in this direction includes [23, 31, 13, 17, 32, 16, 48, 27, 43].”
FBNetV3: Joint Architecture-Recipe Search using Neural Acquisition Function (2020, methodology): “This includes convolutional variants like depthwise convolutions in MobileNet, inverted residual blocks in MobileNetV2; activations such as hswish in MobileNetV3 [14, 31, 13]; and memory-based operations like shift [39] and shufﬂe [24].”
Searching for MobileNetV3 (2019, methodology): “ShiftNet[46] proposes the shift operation interleaved with point-wise convolutions to replace expensive spatial convolutions.”
TSM: Temporal Shift Module for Efficient Video Understanding (2018, methodology): “Address shift, which is a hardware-friendly primitive, has also been exploited for compact 2D CNN design on image recognition tasks [51, 57].”
ChamNet: Towards Efficient Network Design Through Platform-Aware Model Adaptation (2018, methodology): “Compact architecture: Apart from simplifying existing models, handcrafting more efficient building blocks and operators for mobile-friendly architectures can also substantially improve the accuracy-efficiency trade-offs [18, 30].”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “a) Designing efficient NN model architectures: One line of work has focused on optimizing the NN model architecture in terms of its micro-architecture [99, 109, 124, 161, 162, 202, 240, 268] (e.”
Hardware-Guided Symbiotic Training for Compact, Accurate, yet Execution-Efficient LSTM (2019, background): “In addition, a large NN model consumes substantial storage, memory bandwidth, and computational resources that may be too excessive for mobile and embedded devices [9], [10], [11], [12].”
Learning Student Networks via Feature Embedding (2018, background): “[32] presented a parameter-free “shift” operation to alternate vanilla convolutions.”
SqueezeWave: Extremely Lightweight Vocoders for On-device Speech Synthesis (2020, background): “Previous work (Iandola et al., 2016; Wu et al., 2016; 2017; 2018a;b; Yang et al., 2018; Gholami et al., 2018; Wu, 2019; Wu et al., 2019; Dai et al., 2019) have shown that optimizing the neural network architecture can lead to significant efficiency improvement in many applications.”
Temporal Shift Module for Efficient Video Understanding (2018, methodology): “Address shift, which is a hardware-friendly primitive, has also been exploited for compact 2D CNN design on image recognition tasks [45, 50].”
Synetgy: Algorithm-hardware Co-design for ConvNet Accelerators on Embedded FPGAs (2018, methodology): “ efficient models with small model size, low FLOP counts, hardware friendly skip connections, and competitive accuracy. We optimize the network by replacing all 3×3 convolutions with shift operations [11] and 1×1 convolution, enabling us to implement a compute unit customized for 1×1 convolutions fo”
Online Continual Learning for Embedded Devices (2022, neutral): “…(SqueezeNet) (Iandola et al., 2016), group convolutions (ShuffleNet, CondenseNet) (Zhang et al., 2018; Huang et al., 2018), depth-wise separable convolutions (MobileNet-v1, MobileNet-v2) (Howard et al., 2017; Sandler et al., 2018), and point-wise convolutions (ShiftNet) (Wu et al., 2018).”
Efficient Long-Range Attention Network for Image Super-resolution (2022, methodology): “To enlarge the receptive field for more effective local feature extraction, we utilize two shift-conv [56] with a simple ReLU activation between them.”
Efficient Low-Resolution Face Recognition via Bridge Distillation (2020, methodology): “Recently, ShiftFaceNet [57] applies a light-weight network with 0.79M parameters to recognize 224 × 224 face images, thus still costs heavy commutation burden.”
MiniRocket: A Very Fast (Almost) Deterministic Transform for Time Series Classification (2020, methodology): “…(see Sections 3.2.3 and 3.2.4), the optimisations used in MiniRocket bear some resemblance to highly simplified versions of shift-based methods [37], where conventional convolutional kernels are replaced by a combination of 1 × 1 convolutions and spatial shifts in the input, and lookup-based…”
Learning Depth-Guided Convolutions for Monocular 3D Object Detection (2019, methodology): “To reduce the time cost, we employ the shift and element-wise product operators, in which shift [51] is a zero-flop zero-parameter operation, and element-wise product requires little calculation.”
SqueezeSegV3: Spatially-Adaptive Convolution for Efficient Point-Cloud Segmentation (2020, methodology): “Many neural nets have been targeted to achieve efficiency, including SqueezeNet [16,10,54], MobileNets [12,39,11], ShiftNet [55,61], ShuffleNet [63,28], FBNet [53,57], ChamNet [7], MnasNet [44], and EfficientNet [45].”
Dynamic Convolution: Attention Over Convolution Kernels (2019, methodology): “ShiftNet [31] replaces expensive spatial convolution by the shift operation and pointwise convolutions.”