Learning to Learn with Generative Models of Neural Network Checkpoints

By William S. Peebles, Ilija Radosavovic, Tim Brooks, Alexei A. Efros, Jitendra Malik. 2022

100 citations recorded by the source corpus.

We explore a data-driven approach for learning to optimize neural networks. We construct a dataset of neural network checkpoints and train a generative model on the parameters. In particular, our model is a conditional diffusion transformer that, given an initial input parameter vector and a prompted loss, error, or return, predicts the distribution over parameter updates that achieve the desired metric. At test time, it can optimize neural networks with unseen parameters for downstream tasks in just one update. We find that our approach successfully generates parameters for a wide range of loss prompts. Moreover, it can sample multimodal parameter solutions and has favorable scaling properties. We apply our method to different neural network architectures and tasks in supervised and reinforcement learning.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Alexei A. Efros
2. Saining Xie
3. Aleksander Holynski
4. William S. Peebles
5. Trevor Darrell
6. Tim Brooks
7. Abhinav Kumar Gupta
8. Alan L. Yuille

In Statistical and Nonlinear Physics

1. Alexei A. Efros
2. Saining Xie
3. Aleksander Holynski
4. Tim Brooks
5. William S. Peebles

In Meta & Continual Learning

1. Alexei A. Efros

Citations in the corpus

• Scalable Diffusion Models with Transformers (2022, background): “Transformers [60] have replaced domain-specific architectures across language, vision [10], reinforcement learning [5, 25] and meta-learning [39].”
• InstructPix2Pix: Learning to Follow Image Editing Instructions (2022, background): “Diffusion-based generative models Recent advances in diffusion models [59] have enabled state-of-the-art image synthesis [10, 18, 19, 53, 55, 60] as well as generative models of other modalities such as video [21, 58], audio [30], text [35] and network parameters [45].”