OptNet: Differentiable Optimization as a Layer in Neural Networks

By Brandon Amos, J. Kolter. 2017

1,283 citations recorded by the source corpus.

This paper presents OptNet, a network architecture that integrates optimization problems (here, specifically in the form of quadratic programs) as individual layers in larger end-to-end trainable deep networks. These layers encode constraints and complex dependencies between the hidden states that traditional convolutional and fully-connected layers often cannot capture. In this paper, we explore the foundations for such an architecture: we show how techniques from sensitivity analysis, bilevel optimization, and implicit differentiation can be used to exactly differentiate through these layers and with respect to layer parameters; we develop a highly efficient solver for these layers that exploits fast GPU-based batch solves within a primal-dual interior point method, and which provides backpropagation gradients with virtually no additional cost on top of the solve; and we highlight the application of these approaches in several problems. In one notable example, we show that the method is capable of learning to play mini-Sudoku (4x4) given just input and output games, with no a priori information about the rules of the game; this highlights the ability of our architecture to learn hard constraints better than other neural architectures.

Subfields: Computational Mechanics, Numerical Analysis, Statistical and Nonlinear Physics, Model Architectures, Optimization & Training, Mathematical Optimization

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

  1. Fabian Pedregosa
  2. Chunhua Shen
  3. Guosheng Lin
  4. Valentina Zantedeschi
  5. Byron Boots
  6. Vlad Niculae
  7. Matt J. Kusner
  8. Subhransu Maji

In Numerical Analysis

  1. Shane T. Barratt
  2. Steven Diamond
  3. Chunhua Shen
  4. Guosheng Lin
  5. Chi Zhang
  6. Yujun Cai
  7. Vlad Niculae
  8. Matt J. Kusner

In Statistical and Nonlinear Physics

  1. Sergey Levine
  2. Nils Thürey
  3. Aravind Rajeswaran
  4. Vladlen Koltun
  5. Chelsea Finn
  6. Byron Boots
  7. Igor Mordatch
  8. Bistra N. Dilkina

In Model Architectures

  1. Chunhua Shen
  2. J. Kolter
  3. Vladlen Koltun
  4. Guosheng Lin
  5. Jia Deng
  6. Zachary Teed
  7. Shaojie Bai
  8. Raquel Urtasun

In Optimization & Training

  1. Simon Lacoste-Julien
  2. Fabian Pedregosa
  3. Sergey Levine
  4. Sham M. Kakade
  5. Chelsea Finn
  6. Stefano Soatto
  7. Byron Boots
  8. Zeyuan Allen-Zhu

In Mathematical Optimization

  1. Chi Zhang
  2. Chunhua Shen
  3. Yujun Cai
  4. Guosheng Lin
  5. Byron Boots
  6. Simon Lacoste-Julien
  7. J. Kolter
  8. Andreas Krause

Citations in the corpus