Deep Unfolding: Model-Based Inspiration of Novel Deep Architectures

By John R. Hershey, Jonathan Le Roux, Felix Weninger. 2014

492 citations recorded by the source corpus.

Model-based methods and deep neural networks have both been tremendously successful paradigms in machine learning. In model-based methods, problem domain knowledge can be built into the constraints of the model, typically at the expense of difficulties during inference. In contrast, deterministic deep neural networks are constructed in such a way that inference is straightforward, but their architectures are generic and it is unclear how to incorporate knowledge. This work aims to obtain the advantages of both approaches. To do so, we start with a model-based approach and an associated inference algorithm, and \emph{unfold} the inference iterations as layers in a deep network. Rather than optimizing the original model, we \emph{untie} the model parameters across layers, in order to create a more powerful network. The resulting architecture can be trained discriminatively to perform accurate inference within a fixed network size. We show how this framework allows us to interpret conventional networks as mean-field inference in Markov random fields, and to obtain new architectures by instead using belief propagation as the inference algorithm. We then show its application to a non-negative matrix factorization model that incorporates the problem-domain knowledge that sound sources are additive. Deep unfolding of this model yields a new kind of non-negative deep neural network, that can be trained using a multiplicative backpropagation-style update algorithm. We present speech enhancement experiments showing that our approach is competitive with conventional neural networks despite using far fewer parameters.

Subfields: Artificial Intelligence, Model Architectures, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Signal Processing

Citations in the corpus

Convolutional Networks on Graphs for Learning Molecular Fingerprints (2015, background): “Unrolled inference algorithms [9] and others have noted that iterative inference procedures sometimes resemble the feedforward computation of a recurrent neural network.”
Discriminative Embeddings of Latent Variable Models for Structured Data (2016, neutral): “In these existing messages learning work (Hershey et al., 2014; Zheng et al., 2015; Lin et al., 2015), the learning task is still estimating the messages represented graphical models with designed function forms, e.”
Interpretable Recurrent Neural Networks Using Sequential Sparse Recovery (2016, methodology): “In this paper, we use the idea of deep unfolding [9] to make a modiﬁcation to the optimization problem (1), given by (2).”
End-to-end text-dependent speaker verification (2015, background): “The proposed neural network architecture can be thought of as joint optimization of a generative-discriminative hybrid and is in the same spirit as deep unfolding [18] for adaptation.”
Neural Expectation Maximization (2017, background): “An alternative to further empower the network by untying its weights across iterations [11] was not considered for ﬂexibility reasons.”
Structured Prediction Energy Networks (2015, methodology): “Optimization over [0 , 1] L can be performed using projected gradient descent, or entropic mirror descent by normalizing over each coordinate (Beck & Teboulle, 2003).”
Using Deep Neural Networks for Inverse Problems in Imaging: Beyond Analytical Methods (2018, methodology): “LISTA [47] represents a simple case of an unfolding algorithm [48] that aims to combine the advantages of both analytical approaches and neural networks.”
Learning to Detect (2018, background): “In particular, one promising approach to designing deep architectures is by unfolding an existing iterative algorithm [14].”
Deep ADMM-Net for Compressive Sensing MRI (2016, methodology): “The discriminative parameter learning approach has been applied to sparse coding and Markov Random Filed [16, 17, 18, 19].”
Model-Based Deep Learning (2020, methodology): “Examples of Model-aided networks for inference include Deep unfolding [23], also referred to as deep unrolling [9], where a network is trained to perform a fixed number of gradient descent iterations, as well as CausalGANs [24], which embed a known causal graph in a DNN architecture.”
Learning to Optimize: Training Deep Neural Networks for Interference Management (2017, methodology): “The authors of [12], [13] applied similar ideas for nonnegative matrix factorization and the alternating direction method of multipliers (ADMM), respectively.”
Quantum Machine Learning for 6G Communication Networks: State-of-the-Art and Vision for the Future (2019, unknown): “, deep unfolding method in [105].”
Learning to Optimize: A Primer and A Benchmark (2021, methodology): “Examples include clustering [221; 169; 170] and classification [222; 262; 59], phase retrieval [97], RNA second structure prediction [46], speech recognition and source separation [89; 225], remote sensing [130], smart grid [258], graph recovery [182], and photometric stereo estimation [235].”
New Era for Robust Speech Recognition (2017, methodology): “In addition, speaker adaptation and normalization techniques have further mitigated the robustness issues due to speaker variations [7, 8, 12, 19, 20].”
AMP-Inspired Deep Networks for Sparse Linear Inverse Problems (2016, methodology): “For example, convolutional neural networks and stacked denoising autoencoders have been applied to speech enhancement [27], image denoising [28], image deblurring [29], [30], image super resolution [31], 3D imaging [32], compressive imaging [33]– [35], and video compressive sensing [36].”
Algorithm Unrolling: Interpretable, Efficient Deep Learning for Signal and Image Processing (2019, methodology): “[29] 2014 Speech processing Signal channel source separation Nonnegative matrix factorization (NMF) Wang et al.”
Modeling, Inference and Optimization With Composable Differentiable Procedures (2016, background): “[41] and others have noted that iterative inference procedures sometimes resemble the feedforward computation of a recurrent neural network.”
Learned D-AMP: Principled Neural Network based Compressive Image Recovery (2017, methodology): “Moving beyond CS, the unrolling principle has been applied successfully to speech enhancement [29], non-negative matrix factorization applied to music transcription [30], and more.”
End-to-End Learning for Structured Prediction Energy Networks (2017, methodology): “End-to-end learning is also applicable to special-case energy minimization algorithms for graphical models, such as mean-ﬁeld inference and belief propagation (Domke, 2013; Chen et al., 2015; Tompson et al., 2014; Li & Zemel, 2014; Hershey et al., 2014; Zheng et al., 2015).”