Flattened Convolutional Neural Networks for Feedforward Acceleration

By Jonghoon Jin, Aysegul Dundar, Eugenio Culurciello. 2014

264 citations recorded by the source corpus.

We present flattened convolutional neural networks that are designed for fast feedforward execution. The redundancy of the parameters, especially weights of the convolutional filters in convolutional neural networks has been extensively studied and different heuristics have been proposed to construct a low rank basis of the filters after training. In this work, we train flattened networks that consist of consecutive sequence of one-dimensional filters across all directions in 3D space to obtain comparable performance as conventional convolutional networks. We tested flattened model on different datasets and found that the flattened layer can effectively substitute for the 3D filters without loss of accuracy. The flattened convolution pipelines provide around two times speed-up during feedforward pass compared to the baseline model due to the significant reduction of learning parameters. Furthermore, the proposed method does not require efforts in manual tuning or post processing once the model is trained.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Efficiency & Compression

Citations in the corpus

MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications (2017, background): “There has been rising interest in building small and efﬁ-cient neural networks in the recent literature, e.g. [16, 34, 12, 36, 22].”
Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation (2018, methodology): “This operation has been adopted in many recent neural network designs [35, 74, 12, 31, 80, 60, 84].”
ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices (2017, background): “Quantization [31, 27, 39, 45, 44] and factorization [22, 16, 18, 37] are proposed in literature to reduce redundancy in calculations to speed up inference.”
PerforatedCNNs: Acceleration through Elimination of Redundant Convolutions (2015, methodology): “ral convolutional layers applied sequentially, which have a much lower total computational cost. Another direction of research is modiﬁcations of CNN architecture for speed gains (He & Sun, 2015; Jin et al., 2015; Romero et al., 2014). The resulting networks are usually CNNs, meaning that our ap”
Moonshine: Distilling with Cheap Convolutions (2017, background): “In Jin et al. (2015) the authors propose breaking up the general 3D convolution into a set of 3 pointwise convolutions along different axes.”
Review of Image Classification Algorithms Based on Convolutional Neural Networks (2021, background): “[128,129] also show that using separable convolution can reduce the size and computational cost of CNNs.”
ESPNet: Efficient Spatial Pyramid of Dilated Convolutions for Semantic Segmentation (2018, background): “Inception [11,12,13], factorized network [22], ResNext [14], Xception [15], and MobileNets [16]).”
Learning Student Networks via Feature Embedding (2018, background): “[34] presented fully factorized convolutions to accelerate the feedforward of deep networks.”
ExpandNets: Linear Over-parameterization to Train Compact Convolutional Networks (2018, methodology): “In this context, existing approaches can be roughly grouped into two categories: (i) parameter pruning and sharing [7, 9, 14, 16, 31, 32, 38, 55], which aims to remove the least informative parameters, yielding an arbitrary compact network with information loss; and (ii) low-rank matrix factorizatio”
Wide Activation for Efficient and Accurate Image Super-Resolution (2018, background): “Flattened convolution Flattened convolutions [13] consist of consecutive sequence of one-dimensional ﬁlters across all directions in 3D space (lateral, vertical and horizontal) to approximate conventional convolutions.”
Interleaved Group Convolutions for Deep Neural Networks (2017, unknown): “In the spatial extent, small kernels are developed, such as 3 × 3 , 3 × 1 , 1 × 3 [35, 29, 17, 26, 18].”
GhostNet: More Features From Cheap Operations (2019, methodology): “Tensor decomposition [23, 25, 7] reduces the parameters or computation by exploiting the redundancy and low-rank property in weights.”
ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation (2016, background): “Factorizing ﬁlters It has been shown that convolutional weights have a fair amount of redundancy, and each n × n convolution can be decomposed into two smaller ones following each other: one with a n × 1 ﬁlter and the other with a 1 × n ﬁlter [32].”
Convolutional networks for fast, energy-efficient neuromorphic computing (2016, background): “This approach draws inspiration from the spiking neurons and lowprecision synapses of the brain (8) and builds on work showing that deep learning can create networks with constrained connectivity (9), low-precision synapses (10, 11), low-precision neurons (12–14), or both low-precision synapses and ”
ACNet: Strengthening the Kernel Skeletons for Powerful CNN via Asymmetric Convolution Blocks (2019, methodology): “However, as the learned kernels in deep networks have distributed eigenvalues, their intrinsic rank is higher than one in practice, thus applying the transformation directly to the kernels results in significant information loss [18].”
Factorized Convolutional Neural Networks (2016, background): “Low rank decomposition has been extensively explored in various fashions [3][10][25][8][22][11][24] to obtain moderate efficiency improvement.”
MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer (2021, methodology): “Because this layer is computationally expensive, several factorization-based methods have been proposed to make it light-weight and mobile-friendly (e.g., Jin et al., 2014; Chollet, 2017; Mehta et al., 2020).”