Structured Transforms for Small-Footprint Deep Learning

By Vikas Sindhwani, Tara N. Sainath, Sanjiv Kumar. 2015

253 citations recorded by the source corpus.

We consider the task of building compact deep learning pipelines suitable for deployment on storage and power constrained mobile devices. We propose a unified framework to learn a broad family of structured parameter matrices that are characterized by the notion of low displacement rank. Our structured transforms admit fast function and gradient evaluation, and span a rich range of parameter sharing configurations whose statistical modeling capacity can be explicitly tuned along a continuum from structured to unstructured. Experimental results show that these transforms can significantly accelerate inference and forward/backward passes during training, and offer superior accuracy-compactness-speed tradeoffs in comparison to a number of existing techniques. In keyword spotting applications in mobile speech recognition, our methods are much more effective than standard linear low-rank bottleneck layers and nearly retain the performance of state of the art models, while providing more than 3.5-fold compression.

Subfields: Artificial Intelligence, Model Architectures, Efficiency & Compression, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Efficiency & Compression

In Signal Processing

Citations in the corpus

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022, background): “There have been several more general classes of structured matrices proposed in machine learning: Toeplitz-like [81], low-displacement rank [51], quasi-separable [27]).”
MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications (2017, background): “Other reduced computation networks include structured transform networks [28] and deep fried convnets [37].”
Knowledge Distillation: A Survey (2020, background): “This category is further divided into model quantization (Wu et al. 2016), model binarization (Courbariaux et al. 2015), structural matrices (Sindhwani et al. 2015) and parameter sharing (Han et al. 2015; Wang et al. 2019f).”
ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression (2017, neutral): “Low-rank approximation is also widely studied [5, 29].”
Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization (2019, methodology): “We also compared our dynamic sparse reparameterization method to a number of static dense reparameterization techniques, e.g. (Denil et al., 2013; Yang et al., 2014; Moczulski et al., 2015; Sindhwani et al., 2015; Chen et al., 2015; Treis-ter et al., 2018).”
FNet: Mixing Tokens with Fourier Transforms (2021, methodology): “…Zhang et al., 2018), Trans-formers (Choromanski et al., 2020; Tamkin et al., 2020), and MLP layers more generally (Cheng et al., 2015; Moczulski et al., 2016; Sindhwani et al., 2015), we believe our work is the ﬁrst to wholly replace particular neural network sublayers with a Fourier Transform.”
Composition of Deep and Spiking Neural Networks for Very Low Bit Rate Speech Coding (2016, background): “The low-rank factorization and Toeplitz matrices are found to enable significant memory reduction with minor or no reduction in performance [75], [76].”
Discrimination-aware Channel Pruning for Deep Neural Networks (2018, methodology): “To remove such redundancy, low-rank approximation approaches have been widely studied [4, 5, 17, 36].”
BlockDrop: Dynamic Inference Paths in Residual Networks (2017, background): “The need to deploy top-performing deep neural network models on mobile devices motivates techniques that can effectively reduce the storage and computational costs of such networks, including knowledge distillation [19, 40, 4], low-rank factorization [25, 47, 41], filter pruning [30, 36, 32, 57], qu”
Fully-Adaptive Feature Sharing in Multi-Task Networks with Applications in Person Attribute Classification (2016, methodology): “Methods for compressing and accelerating convolutional networks include knowledge distillation [12, 29], low-rank-factorization [14, 39, 33], pruning and quantization [10, 27], structured matrices [6, 35, 9], and dynamic capacity networks [2].”
Data-Dependent Coresets for Compressing Neural Networks with Applications to Generalization Bounds (2018, methodology): “[44] Vikas Sindhwani, Tara Sainath, and Sanjiv Kumar.”
Learning Fast Algorithms for Linear Transforms Using Butterfly Factorizations (2019, methodology): “Fully connected Previous work showed that structured matrix approaches based on the low displacement rank framework, including Toeplitz-like [38], LDR-SD and LDR-TD matrices [40], compare very favorably to other compression approaches.”
An Entropy-based Pruning Method for CNN Compression (2017, methodology): “Early works [3, 33] mainly focus on compressing parameters of the fully-connected layers, which are suffering from great redundancy and are easy to compress.”
Communication-Efficient Edge AI: Algorithms and Systems (2020, background): “…sketching [62], [63], [64], network pruning [65], [66], [67], [27], [68], [69], [70], [71], sparse regularization [72], [73], [74], and structural matrix designing for minimizing communication bandwidth [75], [76], [77], [78], [79], [80], [81], [82], [83] Communication-EfﬁcientEdgeAISystems”
Monarch: Expressive Structured Matrices for Efficient and Accurate Training (2022, methodology): “, Toeplitz-like (Sindhwani et al., 2015), low-displacement rank (Kailath et al.”
Tensorial Neural Networks: Generalization of Neural Networks and Application to Model Compression (2018, background): “TNN are generalization of traditional neural networks: if the input is a vector and operation is matrix-vector multiplication, TNN reduces to multi-layer perceptrons (MLP) ; and if the input is a feature map and operation is convolution, TNN reduces to convolutional neural networks (CNN) .”
Learning compact recurrent neural networks (2016, methodology): “Recently, [10] proposed a new family of parameter sharing schemes for small-footprint deep learning based on structured matrices characterized by the notion of displacement operators [16].”
Personalized speech recognition on mobile devices (2016, methodology): “SVD has elsewhere been shown to be effective for speech processing tasks [4, 6, 7] as have structured transforms [8] and low-rank matrix factorizations [9].”