A Mathematical Theory of Deep Convolutional Neural Networks for Feature Extraction

By Thomas Wiatowski, Helmut Bölcskei. 2015

400 citations recorded by the source corpus.

Deep convolutional neural networks (DCNNs) have led to breakthrough results in numerous practical machine learning tasks, such as classification of images in the ImageNet data set, control-policy-learning to play Atari games or the board game Go, and image captioning. Many of these applications first perform feature extraction and then feed the results thereof into a classifier. The mathematical analysis of DCNNs for feature extraction was initiated by Mallat, 2012. Specifically, Mallat considered so-called scattering networks based on a wavelet transform followed by the modulus non-linearity in each network layer, and proved translation invariance (asymptotically in the wavelet scale parameter) and deformation stability of the corresponding feature extractor. This paper complements Mallat’s results by developing a theory that encompasses general convolutional transforms, or in more technical parlance, general semi-discrete frames (including Weyl–Heisenberg filters, curvelets, shearlets, ridgelets, wavelets, and learned filters), general Lipschitz-continuous non-linearities (e.g., rectified linear units, shifted logistic sigmoids, hyperbolic tangents, and modulus functions), and general Lipschitz-continuous pooling operators emulating, e.g., sub-sampling and averaging. In addition, all of these elements can be different in different network layers. For the resulting feature extractor, we prove a translation invariance result of vertical nature in the sense of the features becoming progressively more translation-invariant with increasing network depth, and we establish deformation sensitivity bounds that apply to signal classes such as, e.g., band-limited functions, cartoon functions, and Lipschitz functions.

Subfields: Applied Mathematics, Computer Vision and Pattern Recognition, Geophysics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Applied Mathematics

In Computer Vision and Pattern Recognition

In Geophysics

Citations in the corpus

Robust Large Margin Deep Neural Networks (2016, background): “Scattering transform—a convolutional DNN like transform, which is based on the wavelet transform and pointwise nonlinearities—provides insights into translation invariance and stability to deformations of convolutional DNNs [13]–[15].”
Deep D-Bar: Real-Time Electrical Impedance Tomography Imaging With Deep Neural Networks (2017, background): “Additionally, as discussed in [31], pooling layers leads to translational invariance, which is important to reduce locational bias in the reconstruction process and detect injuries not present in the training set.”
Concolic Testing for Deep Neural Networks (2018, methodology): “Lipschitz continuity is semantic, specific to DNNs, and has been shown to be closely related to the theoretical understanding of convolutional DNNs [29] and the robustness of both DNNs [20, 30] and Generative Adversarial Networks [1].”
Convolutional Neural Network Architectures for Signals Supported on Graphs (2018, neutral): “[u ]n /|n | [27].”
Group Invariance, Stability to Deformations, and Complexity of Deep Convolutional Representations (2017, background): “We note that an extension of scattering networks results to more general convolutional networks was previously given by Wiatowski and B¨olcskei (2018); however, their guarantees on deformations do not improve on the inherent stability properties of the considered signal, and their study does not…”
Interpretable Convolutional Neural Networks via Feedforward Design (2018, background): “Recent studies on CNNs include: visualization of ﬁlter responses at various layers [6, 7, 8], scattering networks [9, 10, 11], tensor analysis [12], generative modeling [13], relevance propagation [14], Taylor decomposition [15], multi-layer convolutional sparse modeling [16] and over-parameterized…”
Prevalence of neural collapse during the terminal phase of deep learning training (2020, background): “Later, (35–37), complemented (33) with additional theory covering a larger range of mathematically-derived features, nonlinearities, and pooling operations – again designed to suppress within-class variability.”
Generalization Error of Invariant Classifiers (2016, background): “See also (Sifre and Mal-lat, 2013; Wiatowski and B¨olcskei, 2015).”
A Group-Theoretic Framework for Data Augmentation (2019, background): “Related work by Mallat, Bölcskei and others (e.g., Mallat, 2012; Bruna and Mallat, 2013; Wiatowski and Bölcskei, 2018; Anselmi et al., 2019) tries to explain how CNNs extract features, using ideas from harmonic analysis.”
Optimal approximation of piecewise smooth functions using deep ReLU neural networks (2017, methodology): “ces. For instance, an image classiﬁer should be translation invariant, scaling invariant, invariant to small smooth deformations, and invariant to small changes in color, brightness, or contrast; see [39, 61]. A way to model such a function class is by resorting to a two-step procedure: The occurren”
Deep Semi-Supervised Anomaly Detection (2019, background): “The study of the theoretical foundations of deep learning is an active and ongoing research effort [43, 66, 14, 18, 46, 52, 73, 1, 5, 6, 70, 36].”
Deep Convolutional Neural Networks for Image Classification: A Comprehensive Review (2017, background): “Wiatowski and Bölcskei (2015) persisted with the mathematical analysis of the features extracted by DCNNs, and they theoretically established deformation stability and vertical translation invariance.”
On Lipschitz Bounds of General Convolutional Neural Networks (2018, methodology): “The idea of using Bessel bounds is also addressed in [9] where the authors compute the Bessel bounds of each layer of the AlexNet, and in [4] where the authors set Bm ≤ 1 to make the CNN a 1-Lipschitz map.”