JFB: Jacobian-Free Backpropagation for Implicit Networks

By Samy Wu Fung, Howard Heaton, Qiuwei Li, Daniel Mckenzie, Stanley J. Osher, Wotao Yin. 2021

139 citations recorded by the source corpus.

A promising trend in deep learning replaces traditional feedforward networks with implicit networks. Unlike traditional networks, implicit networks solve a fixed point equation to compute inferences. Solving for the fixed point varies in complexity, depending on provided data and an error tolerance. Importantly, implicit networks may be trained with fixed memory costs in stark contrast to feedforward networks, whose memory requirements scale linearly with depth. However, there is no free lunch --- backpropagation through implicit networks often requires solving a costly Jacobian-based equation arising from the implicit function theorem. We propose Jacobian-Free Backpropagation (JFB), a fixed-memory approach that circumvents the need to solve Jacobian-based equations. JFB makes implicit networks faster to train and significantly easier to implement, without sacrificing test accuracy. Our experiments show implicit networks trained with JFB are competitive with feedforward networks and prior implicit networks given the same number of parameters.

Subfields: Civil and Structural Engineering, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Civil and Structural Engineering

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Optimization & Training

Citations in the corpus

Multiset-Equivariant Set Prediction with Approximate Implicit Differentiation (2021, background): “Fung et al. (2022) provide motivations for this approximation.”
A-ViT: Adaptive Tokens for Efficient Vision Transformer (2021, background): “One way is to change the architecture of the model to support adaptive computations [4,14,15, 18, 25, 27, 30, 37, 42, 51, 54].”