Gradients are Not All You Need

By Luke Metz, C. Freeman, Sam Schoenholz, Tal Kachman. 2021

111 citations recorded by the source corpus.

Differentiable programming techniques are widely used in the community and are responsible for the machine learning renaissance of the past several decades. While these methods are powerful, they have limits. In this short report, we discuss a common chaos based failure mode which appears in a variety of differentiable circumstances, ranging from recurrent neural networks and numerical physics simulation to training learned optimizers. We trace this failure to the spectrum of the Jacobian of the system under study, and provide criteria for when a practitioner might expect this failure to spoil their differentiation based optimization algorithms.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

In Optimization & Training

Citations in the corpus

Learning to Learn with Generative Models of Neural Network Checkpoints (2022, background): “This allow us to train with standard generative modeling techniques instead of reinforcement learning or unrolled optimization which can be unstable (Metz et al., 2021).”
Human-Timescale Adaptation in an Open-Ended Task Space (2023, background): “, 2018), which can be efficient but often suffer from instability and myopia (Flennerhag et al., 2022; Metz et al., 2021; Vuorio et al., 2021), and latent-variable based approaches (Finn et al.”