Convolutional Neural Networks Analyzed via Convolutional Sparse Coding

By Vardan Papyan, Yaniv Romano, Michael Elad. 2016

309 citations recorded by the source corpus.

Convolutional neural networks (CNN) have led to many state-of-the-art results spanning through various fields. However, a clear and profound theoretical understanding of the forward pass, the core algorithm of CNN, is still lacking. In parallel, within the wide field of sparse approximation, Convolutional Sparse Coding (CSC) has gained increasing attention in recent years. A theoretical study of this model was recently conducted, establishing it as a reliable and stable alternative to the commonly practiced patch-based processing. Herein, we propose a novel multi-layer model, ML-CSC, in which signals are assumed to emerge from a cascade of CSC layers. This is shown to be tightly connected to CNN, so much so that the forward pass of the CNN is in fact the thresholding pursuit serving the ML-CSC model. This connection brings a fresh view to CNN, as we are able to attribute to this architecture theoretical claims such as uniqueness of the representations throughout the network, and their stable estimation, all guaranteed under simple local sparsity conditions. Lastly, identifying the weaknesses in the above pursuit scheme, we propose an alternative to the forward pass, which is connected to deconvolutional, recurrent and residual networks, and has better theoretical guarantees.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

RotDCF: Decomposition of Convolutional Filters for Rotation-Equivariant Deep Networks (2018, methodology): “[26] interpreted CNN as an iterated convolutional sparse coding machine, and in this view, the factorized ﬁlters should correspond to a “dictionary of the dictionary” as in [30].”
Deep Image Prior (2017, background): “The connections between ConvNets and convolutional sparse coding run even deeper and are investigated in [24] in the context of recognition networks, and more recently in [25], where a single-layer convolutional sparse coding is proposed for reconstruction tasks.”
Simple and Deep Graph Convolutional Networks (2020, methodology): “Graph convolutional networks (GCNs) (Kipf & Welling, 2017) generalize convolutional neural networks (CNNs) (Le-Cun et al., 1995) to graph-structured data.”
DCFNet: Deep Neural Network with Decomposed Convolutional Filters (2018, background): “Making a connection to dictionary learning (Aharon et al., 2006), (Papyan et al., 2016) studied deep networks in form of a cascade of convolutional sparse coding layers with theoretical analysis.”
When and How Can Deep Generative Models be Inverted? (2020, background): “Neural networks and sparse representation: In the grand search for a profound theoretical understanding for deep learning, a series of papers suggested a connection between neural networks and sparse coding [20, 25, 7, 24, 21, 27].”
Solving inverse problems using data-driven models (2019, background): “Several recent papers have discussed the application of deep learning in forward problems: see Khoo, Lu and Ying (2017), Raissi and Karniadakis (2017), Sirignano and Spiliopoulos (2017), Tompson, Schlachter, Sprechmann and Perlin (2017), E, Han and Jentzen (2017) and Wu, Zhang, Shen and Zhai (2018).”
Deep Convolutional Framelets: A General Deep Learning Framework for Inverse Problems (2017, background): “For instance, Papyan et al [56] showed that once ReLU nonlinearity is employed, the forward pass of a network can be interpreted as a deep sparse coding algorithm.”
Prevalence of neural collapse during the terminal phase of deep learning training (2020, background): “In the deepnet literature, the authors of (44–48) studied the stability of deepnets to adversarial examples.”
Deep Networks from the Principle of Rate Reduction (2020, background): “The sparse coding perspective of Gregor & LeCun (2010) has later been extended to recurrent and convolutional networks for serial data, e.g. Wisdom et al. (2016); Papyan et al. (2016); Sulam et al. (2018); Monga et al. (2019).”
Accelerating Convolutional Neural Networks via Activation Map Compression (2018, background): “Recently, connections between CNN’s and Convolutional Sparse Coding (CSC) have been drawn [46, 58].”
RadioUNet: Fast Radio Map Estimation With Convolutional Neural Networks (2019, methodology): “eep learning algorithm receives inputs and produces outputs. The algorithm is written down, or unfolded in the deep learning jargon, as the end-to-end function that transforms the input to the output [33, 34]. This function is sometimes called the network. Unfolding is done by composing the dierent ”
Learning to Optimize: A Primer and A Benchmark (2021, methodology): “[165; 2] extended solving (convolutional) LASSO in a multilayer setting, and an adapted version of ISTA was unrolled in [194].”
Interpretable Convolutional Filters with SincNet (2018, background): “RNN can be employed to capture the temporal evolution of the speech signal [14, 15, 16, 17], while CNNs, thanks to their weight sharing, local filters, and pooling networks are normally employed to extract robust and invariant representations [18].”
Precision Learning: Towards Use of Known Operators in Neural Networks (2017, result): “Similar observations were already done for CNNs in [23].”
Using Deep Neural Networks for Inverse Problems in Imaging: Beyond Analytical Methods (2018, methodology): “Jain and Seung [13] (2008) Denoising Five-layer CNN; trained using greedy layer-wise approach”
Algorithm Unrolling: Interpretable, Efficient Deep Learning for Signal and Image Processing (2019, methodology): “[60] interpret CNNs as executing finite iterations of the Multi-Layer Convolutional Sparse Coding (MLCSC) algorithm.”
Mathematics of Deep Learning (2017, background): “This is related to the recent work in [65] that claims that deep networks may be viewed a sparse coding procedure leading to guarantees on the uniqueness of the representation calculated by the network and its stability.”