Spectral Representations for Convolutional Neural Networks

By Oren Rippel, Jasper Snoek, Ryan P. Adams. 2015

404 citations recorded by the source corpus.

Discrete Fourier transforms provide a significant speedup in the computation of convolutions in deep learning. In this work, we demonstrate that, beyond its advantages for efficient computation, the spectral domain also provides a powerful representation in which to model and train convolutional neural networks (CNNs). We employ spectral representations to introduce a number of innovations to CNN design. First, we propose spectral pooling, which performs dimensionality reduction by truncating the representation in the frequency domain. This approach preserves considerably more information per parameter than other pooling strategies and enables flexibility in the choice of pooling output dimensionality. This representation also enables a new form of stochastic regularization by randomized modification of resolution. We show that these methods achieve competitive results on classification and approximation tasks, without using any dropout or max-pooling. Finally, we demonstrate the effectiveness of complex-coefficient spectral parameterization of convolutional filters. While this leaves the underlying model unchanged, it results in a representation that greatly facilitates optimization. We observe on a variety of popular CNN configurations that this leads to significantly faster convergence during training.

Subfields: Artificial Intelligence, Computational Mechanics, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

End-to-end Optimized Image Compression (2016, background): “We found this to be slightly more effective in speeding up the convergence than discrete Fourier transform (DFT) parameterization (Rippel, Snoek, and Adams, 2015).”
Recent advances in convolutional neural networks (2015, background): “It is important to choose an appropriate loss functions for a speciﬁc task.”
Conditional Time Series Forecasting with Convolutional Neural Networks (2017, methodology): “…for longer time series the prediction of the autoregressive model may be sped up by implementing a recent variation that exploits the memorization structure of the network, see [21] or speeding up the convolutions by working in the frequency domain emloying Fourier transforms as in [19], [22].”
Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Units (2016, neutral): “Figure 4 plots the invariance scores for networks trained on CIFAR-10, CIFAR-100, and ImageNet respectively.”
Learning SO(3) Equivariant Representations with Spherical CNNs (2017, background): “The high performance of these methods is in part due to the use of large pre-trained 2D CNNs (on ImageNet, for instance).”
Real-Time Adaptive Image Compression (2017, neutral): “The spectral representations of CNN activations have also been investigated by Rippel et al. (2015) to enable processing across a spectrum of scales, but this approach does not enable FIR processing as does wavelet analysis.”
Deep Roots: Improving CNN Efficiency with Hierarchical Filter Groups (2016, background): “Most previous work on reducing the computational complexity of CNNs has focused on approximating convolutional filters in the spatial (as opposed to the channel) domain, either by using low-rank approximations [9]–[13], or Fourier transform based convolution [14], [15].”
Frequency Domain Compact 3D Convolutional Neural Networks (2020, methodology): “Discrete Fourier Transform (DFT [27]) and Discrete Cosine Transform (DCT [1]).”
Recent Advances in Convolutional Neural Network Acceleration (2018, neutral): “Rippel [157] Xeon Phi coprocessor Pooling in frequency domain Yes”
Deep Complex Networks (2017, methodology): “Recently, Rippel et al. [2015] leveraged the Fourier spectral representation for convolutional neural networks, providing a technique for parameterizing convolution kernel weights in the spectral domain, and performing pooling on the spectral representation of the signal.”
Deep Convolutional Neural Networks for Image Classification: A Comprehensive Review (2017, methodology): “Their method provided a viable solution to the loss of spatial information associated with max pooling (Ranzato et al., 2007; Scherer et al., 2010; Szegedy, Liu, et al., 2015; Rippel et al., 2015) and a new form of stochastic regularization similar to a Dropout (Hinton et al.”
DeepSense: A Unified Deep Learning Framework for Time-Series Mobile Sensing Data Processing (2016, methodology): “We then apply Fourier transform to each element inW , because the frequency domain contains better local frequency patterns that are independent of how time-series data is organized in the time domain [36].”
Fast Fourier Convolution (2020, background): “For example, [23] proposed spectral pooling that performs dimensionality reduction by truncating the representation in the frequency domain.”
3D object classification and retrieval with Spherical CNNs (2017, background): “The high performance of these methods is in part due to the use of large pre-trained 2D CNNs (on ImageNet, for instance).”
Graph-based Isometry Invariant Representation Learning (2017, methodology): “Further, (Mathieu et al., 2014; Rippel et al., 2015) propose to use convolutional filters in Fourier domain to reduce complexity.”
Pooling Methods in Deep Neural Networks, a Review (2020, background): “[57] introduced a new pooling scheme by including an idea of dimensionality reduction by cropping the representation of the input in the frequency domain.”