Implicit Deep Learning

By Laurent El Ghaoui, Fangda Gu, Bertrand Travacca, Armin Askari, Alicia Y. Tsai. 2019

221 citations recorded by the source corpus.

Implicit deep learning prediction rules generalize the recursive rules of feedforward neural networks. Such rules are based on the solution of a fixed-point equation involving a single vector of hidden features, which is thus only implicitly defined. The implicit framework greatly simplifies the notation of deep learning, and opens up many new possibilities, in terms of novel architectures and algorithms, robustness analysis and design, interpretability, sparsity, and network architecture optimization.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Vladlen Koltun
2. J. Kolter
3. Shaojie Bai
4. Simon Lacoste-Julien
5. Fabian Pedregosa
6. Wotao Yin
7. Sergey Levine
8. Cees G. M. Snoek

In Statistical and Nonlinear Physics

1. Simon Lacoste-Julien
2. Vladlen Koltun
3. Samy Wu Fung
4. Heng Chang
5. J. Kolter
6. Cees G. M. Snoek
7. Wenwu Zhu
8. Jean-Philippe Vert

In Model Architectures

1. Vladlen Koltun
2. Shaojie Bai
3. J. Kolter
4. Stanley J. Osher
5. Wotao Yin
6. Samy Wu Fung
7. Howard Heaton
8. Daniel Mckenzie

Citations in the corpus

• Multiset-Equivariant Set Prediction with Approximate Implicit Differentiation (2021, background): “The implicit function theorem allows for seamless integration of optimization problems into larger end-to-end trainable neural networks (Gould et al., 2016; Amos & Kolter, 2017; Blondel et al., 2021; El Ghaoui et al., 2021).”
• Deep Equilibrium Models (2019, methodology): “Concurrent work [20] also looks at such implicit layers in a broad sense and focuses on training small models via Lagrangian methods; a combination of these approaches with the DEQ model is a promising avenue for future work.”
• Multiscale Deep Equilibrium Models (2020, background): “Implicit approaches to network design have recently attracted renewed interest [20, 24].”
• JFB: Jacobian-Free Backpropagation for Implicit Networks (2021, methodology): “Allowing SΘ to be linear yields the model proposed in (Ghaoui et al. 2019).”
• Implicit Graph Neural Networks (2020, background): “This notion has been introduced in (El Ghaoui et al., 2020) for ordinary implicit models.”
• Efficient and Modular Implicit Differentiation (2021, background): “…line of active research involving differentiation of optimization problem solutions are optimization layers [55, 6, 65, 31, 46], which can be used to encourage structured outputs, and implicit deep networks [8, 36, 43, 44, 73], which have a smaller memory footprint than backprop-trained networks.”