Generic Methods for Optimization-Based Modeling

By Justin Domke. 2012

418 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Mathematical Optimization

Citations in the corpus

Structured Attention Networks (2017, methodology): “Many optimization algorithms (e.g. gradient descent, LBFGS) are also differentiable (Domke, 2012; Maclaurin et al., 2015), and have been used as output layers for structured prediction in energy-based models (Belanger & McCallum, 2016; Wang et al., 2016).”
Learning to Optimize Neural Nets (2017, methodology): “…have been proposed, such those based on Bayesian optimization (Hutter et al., 2011; Bergstra et al., 2011; Snoek et al., 2012; Swersky et al., 2013; Feurer et al., 2015), random search (Bergstra & Bengio, 2012) and gradient-based optimization (Bengio, 2000; Domke, 2012; Maclaurin et al., 2015).”
Gradient-based Hyperparameter Optimization through Reversible Learning (2015, methodology): “Applying RMD to hyperparameter optimization was proposed by Bengio (2000) and Baydin & Pearlmutter (2014), and applied to small problems by Domke (2012).”
Meta-Learning With Differentiable Convex Optimization (2019, methodology): “Domke [6] presented a generic method based on unrolling gradient-descent for a fixed number of steps and automatic differentiation to compute gradients.”
Dataset Distillation (2018, methodology): “Our work bears similarity with gradient-based hyperparameter optimization techniques, which compute the gradient of hyperparameter w.r.t. the ﬁnal validation loss by reversing the entire training procedure (Bengio, 2000; Domke, 2012; Maclaurin et al., 2015; Pedregosa, 2016).”
Hyperparameter optimization with approximate gradient (2016, methodology): “This is the iterative differentiation approach from (Domke, 2012), using the same inner-optimization algorithm as H OAG .”
Towards Poisoning of Deep Learning Algorithms with Back-gradient Optimization (2017, methodology): “However, as these problems are always solved to a finite accuracy, it may happen that the gradient ∇x cA is not sufficiently precise, especially if convergence thresholds are too loose [14, 22].”
Understanding and Robustifying Differentiable Architecture Search (2019, background): “Another strategy is to approximate the inner problem with a dynamical system (Domke, 2012; Maclaurin et al., 2015; Franceschi et al., 2017; 2018), where the optimization dynamics could, e.”
Forward and Reverse Gradient-Based Hyperparameter Optimization (2017, background): “Domke (2012) speciﬁcally considered running an iterative algorithm, like gradient descent or momentum, for a given number of steps, and subsequently computing the gradient of the validation error by a back-optimization al-gorithm.”
Learning to Optimize (2016, methodology): “Most methods [13, 4, 24, 25, 9] rely on sequential model-based Bayesian optimization [22, 6], while others adopt a random search approach [3] or use gradient-based optimization [2, 8, 21].”
OptNet: Differentiable Optimization as a Layer in Neural Networks (2017, background): “Indeed, the general idea of solving restricted classes of optimization problem using neural networks goes back many decades (Kennedy & Chua, 1988; Lillo et al., 1993), but has seen a number of advances in recent years.”
On Differentiating Parameterized Argmin and Argmax Problems with Application to Bi-level Optimization (2016, methodology): “…found application in machine learning and computer vision where they have been applied to parameter and hyper-parameter learning [Do et al., 2007, Domke, 2012, Klatzer and Pock, 2015], image denoising [Samuel and Tappen, 2009, Ochs et al., 2015], and most recently, video activity recognition…”
Reward Augmented Maximum Likelihood for Neural Structured Prediction (2016, background): “Most relevant are ideas for improving approximate maximum likelihood training for intractable models by passing the gradient calculation through an approximate inference procedure [10, 28].”
Structured Prediction Energy Networks (2015, background): “…to y (Amos & Kolter, 2016), but not necessarily the model parameters, and training methods that backpropa-gate through gradient-based prediction (Domke, 2012). where: e 1 i = (¯ y ti, 1 ) (cid:62) A ¯ y ti, 1 + C (cid:62) ¯ y ti, 1 and e 0 i = (¯ y ti, 0 ) (cid:62) A ¯ y ti, 0 + C (cid:62) ¯ y…”
Input Convex Neural Networks (2016, methodology): “The method is also more challenging to integrate with some learning procedures, as we often need to differentiate through an entire chain of the gradient descent algorithm (Domke, 2012).”
An Ensemble of Epoch-Wise Empirical Bayes for Few-Shot Learning (2019, methodology): “pproach, we meta-learn hyperparameters by a simple and elegant gradient descent method, without additional manual labor. Related methods using gradient descent mostly work for single network training [4, 10, 33, 32, 13, 34]. While, we aim to learn a sequence of hyperparameters for multiple base-lear”
Gradient Descent: The Ultimate Optimizer (2019, background): “Most existing work on gradient-based hyperparameter optimization (Bengio, 2000; Domke, 2012; Maclaurin et al., 2015; Pedregosa, 2016; Franceschi et al., 2017) has focused on computing hyperparameter gradients after several iterations of training, which is computationally expensive because of the…”