Deep Declarative Networks
By Stephen Gould, Richard I. Hartley, Dylan Campbell. 2019
125 citations recorded by the source corpus.
We explore a class of end-to-end learnable models wherein data processing nodes (or network layers) are defined in terms of desired behavior rather than an explicit forward function. Specifically, the forward function is implicitly defined as the solution to a mathematical optimization problem. Consistent with nomenclature in the programming languages community, we name these models deep declarative networks. Importantly, it can be shown that the class of deep declarative networks subsumes current deep learning models. Moreover, invoking the implicit function theorem, we show how gradients can be back-propagated through many declaratively defined data processing nodes thereby enabling end-to-end learning. We discuss how these declarative processing nodes can be implemented in the popular PyTorch deep learning software library allowing declarative and imperative nodes to co-exist within the same network. We also provide numerous insights and illustrative examples of declarative nodes and demonstrate their application for image and point cloud classification tasks.
Subfields: Artificial Intelligence, Materials Chemistry, Statistical and Nonlinear Physics
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Andy Zeng
- Jonathan Tompson
- Corey Lynch
- Peter R. Florence
- Ayzaan Wahid
- Igor Mordatch
- Johannes von Oswald
In Statistical and Nonlinear Physics
- Johannes von Oswald
- Igor Mordatch
- Jonathan Tompson
- João Sacramento
- Corey Lynch
- Andy Zeng
- Peter R. Florence
Citations in the corpus
- Implicit Behavioral Cloning (2021, background): “Many other works have investigated using the notion of implicit functions in learning, including works that investigate implicit layers [39, 40, 41, 42, 43].”
- Transformers learn in-context by gradient descent (2022, background): “First, to exceed upon a single step of gradient descent in every self-attention layer it could be advantageous to incorporate so called declarative nodes (Amos & Kolter, 2017; Bai et al., 2019; Gould et al., 2021; Zucchet & Sacramento, 2022) into Transformer architectures.”