Deep Fried Convnets

By Zichao Yang, Marcin Moczulski, Misha Denil, Nando de Freitas, Alex Smola, Le Song, Ziyu Wang. 2014

274 citations recorded by the source corpus.

The fully-connected layers of deep convolutional neural networks typically contain over 90% of the network parameters. Reducing the number of parameters while preserving predictive performance is critically important for training big models in distributed systems and for deployment in embedded devices. In this paper, we introduce a novel Adaptive Fastfood transform to reparameterize the matrix-vector multiplication of fully connected layers. Reparameterizing a fully connected layer with d inputs and n outputs with the Adaptive Fastfood transform reduces the storage and computational costs costs from O(nd) to O(n) and O(n log d) respectively. Using the Adaptive Fastfood transform in convolutional networks results in what we call a deep fried convnet. These convnets are end-to-end trainable, and enable us to attain substantial reductions in the number of parameters without affecting prediction accuracy on the MNIST and ImageNet datasets.

Subfields: Artificial Intelligence, Computational Mechanics, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

HyperNetworks (2016, background): “…interacting with another network is central to the work of (Jaderberg et al., 2016; Andrychowicz et al., 2016), and especially (Denil et al., 2013; Yang et al., 2015; Bertinetto et al., 2016; De Brabandere et al., 2016), where certain parameters in a convolutional network are predicted by another…”
Measuring the Intrinsic Dimension of Objective Landscapes (2018, background): “( v ) There is a beautiful array of papers on compressing networks such that they also achieve computational savings during the forward pass (Wen et al., 2016; Han et al., 2016; Yang et al., 2015); subspace training does not speed up execution time during inference.”
MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications (2017, background): “Other reduced computation networks include structured transform networks [28] and deep fried convnets [37].”
Learning both Weights and Connections for Efficient Neural Network (2015, background): “Deep Fried Convnets [29] worked on fully connected layers only and reduced the parameters by less than 4×.”
Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding (2015, background): “Deep Fried Convnets(Yang et al., 2014) worked on fully connected layers and reduced the parameters by less than 4 × .”
Convolution by Evolution: Differentiable Pattern Producing Networks (2016, methodology): “This allows compression of the weights of convolutional neural networks in order to make them computationally more eﬃ-cient [30, 11].”
Compressing Convolutional Neural Networks in the Frequency Domain (2015, methodology): “[62] adopt the fastfood transformation [32] for compressing the fully connected layers, and Kim et al.”
Deep Kernel Learning (2015, methodology): “In a frequentist setting, Yang et al. (2014) combine convolutional networks, with parameters pre-trained on ImageNet, with a scalable Fastfood (Le et al., 2013) expansion for the RBF kernel applied to the ﬁnal layer.”
Exploration by Random Network Distillation (2018, neutral): “Features of randomly initialized neural networks have been extensively studied in the context of supervised learning (Rahimi & Recht, 2008; Saxe et al., 2011; Jarrett et al., 2009; Yang et al., 2015).”
Dynamic Network Surgery for Efficient DNNs (2016, methodology): “’s method [9] and the adaptive fastfood transform method [21].”
Unitary Evolution Recurrent Neural Networks (2015, methodology): “(Yang et al., 2015) construct a real valued, non-orthogonal matrix using a similar parameterization with the motivation of parameter reduction by an order of magnitude on an industrial sized network.”
Compact Bilinear Pooling (2015, background): “Deep Fried Convnets [40] aims to reduce the number of parameters in the fully connected layer, which usually accounts for 90% of parameters.”
NetAdapt: Platform-Aware Neural Network Adaptation for Mobile Applications (2018, background): “Layer-decomposition-based algorithms [13, 23] exploit matrix decomposition to reduce the number of operations.”
Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Units (2016, methodology): “6M parameters (CReLU + all) outperforms FastFood-32-AD (FriedNet) (Yang et al., 2015) and Pruned AlexNet (PrunedNet) (Han et al.”
Scalable Kernel Methods via Doubly Stochastic Gradients (2014, background): “ling [47] which enjoys faster convergence rate with fewer generated features. Meanwhile, by back-propogation trick, we could rene the random features by adapting their weights for better performance [36]. Acknowledgement M.B. is supported in part by NSF grant CCF-1101283, AFOSR grant FA9550-09-1-053”
Diversity Networks (2015, background): “As these layers are typically responsible for the large majority of the CNNs’ memory footprint (Yang et al., 2014), Divnet is particularly well suited for such networks.”
NISP: Pruning Networks Using Neuron Importance Score Propagation (2017, methodology): “[37] applied the “Fastfood” transform to reparameterize the matrix-vector multiplication of FC layers.”
Capacity and Trainability in Recurrent Neural Networks (2016, positive): “For instance, previous results on model compression – by reducing the number of parameters (Yang et al., 2015), or by reducing the bit depth of parameters (Hubara et al., 2016) – might lead one to predict that different architectures use parameters with vastly different efﬁciencies, and that task…”