Efficient multiple hyperparameter learning for log-linear models

By Chuong B. Do, Chuan-Sheng Foo, Andrew Y. Ng. 2007

120 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Hyperparameter optimization with approximate gradient (2016, methodology): “As convergence criterion, we consider the norm of the (scaled) gradient mapping , a generalization of the gradient that ensures convergence to a stationary point (Beck and Teboulle, 2009, §1.4.4).”
Towards Poisoning of Deep Learning Algorithms with Back-gradient Optimization (2017, methodology): “lving one linear system per parameter. These aspects make it prohibitive to assess the effectiveness of poisoning attacks in a variety of practical settings. To mitigate these issues, as suggested in [13, 14, 20, 22], one can apply conjugate gradient descent to solve a simpler linear system, obtaine”
On Differentiating Parameterized Argmin and Argmax Problems with Application to Bi-level Optimization (2016, methodology): “…problems have found application in machine learning and computer vision where they have been applied to parameter and hyper-parameter learning [Do et al., 2007, Domke, 2012, Klatzer and Pock, 2015], image denoising [Samuel and Tappen, 2009, Ochs et al., 2015], and most recently, video activity…”
Evolutionary algorithms for hyperparameter tuning on neural networks models (2014, background): “However, as the number of hyperparameters becomes large these techniques are inviable because their computational cost (Do, Foo, and Ng 2008).”
Learning optimized MAP estimates in continuously-valued MRF models (2009, methodology): “Previous authors have pointed out two important points regarding Equation 6 [3, 15].”
Bayesian models for Large-scale Hierarchical Classification (2012, background): “For instance, Empirical Bayes requires the maximization of marginal likelihood which is difficult to compute in hierarchical logistic models [9] in general, and cross-validation requires reducing the number of free parameters for computational reasons, potentially losing the flexibility to capture t”
Bilevel Optimization with Nonsmooth Lower Level Problems (2015, methodology): “A clever way of setting parentheses avoids explicit inversion of the Hessian matrix [22].”
Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameters (2015, methodology): “…the late 1990s; however, these methods either require computation of the inverse Hessian (Larsen et al., 1998; Bengio, 2000; Chen and Hagan, 1999; Foo et al., 2008), or propagating gradients 1Borrowing the expression from Maclaurin et al. (2015), we refer to the model parameters customarily…”
End-to-End Learning for Structured Prediction Energy Networks (2017, unknown): “The implicit function theorem offers an alternative framework for training energy-based predictors (Foo et al., 2008; Samuel & Tappen, 2009).”
Differentiation of Blackbox Combinatorial Solvers (2019, methodology): “However, the function f ( y(w) ) is a result of a minimization process and it is known that for smooth spaces Y there are techniques for such “differentiation through argmin” [2, 3, 11, 13, 38, 40].”
Modeling, Inference and Optimization With Composable Differentiable Procedures (2016, background): “[29] showed that gradients of regularization parameters are available in closed form when training has converged exactly to a local minimum.”
Learning End-to-end Video Classification with Rank-Pooling (2016, background): “It has recently gained interest in the machine learning community in the context of hyperparameter learning (Klatzer & Pock, 2015; Do et al., 2007) and in the computer vision community in the context of image denoising (Domke, 2012; Kunisch & Pock, 2013).”
RNA secondary structure prediction using deep learning with thermodynamic integration (2020, methodology): “The maximum expected accuracy (MEA)-based secondary structure predictions, CONTRAfold and CentroidFold, can control PPV and SEN through the parameter γ, and therefore, their accuracies are shown as the curves for multiple γ ∈ {2n | −5 ≤ n ≤ 10}.”
Generic Methods for Optimization-Based Modeling (2012, unknown): “in the context of hyper-parameter learning (Do et al., 2007).”
Reviving and Improving Recurrent Back-Propagation (2018, methodology): “For example, an efficient gradient-based method to learn the hyperparameters of log-linear models is provided in (Foo et al., 2008) where the core contribution is to use the implicit differentiation trick to compute the gradient of the optimized inference solution w.”
Self-Tuning Networks: Bilevel Optimization of Hyperparameters using Structured Best-Response Functions (2019, methodology): “Similar approaches have been used for hyperparameter optimization in log-linear models (Foo et al., 2008), kernel selection (Chapelle et al.”
Scalable Gaussian process-based transfer surrogates for hyperparameter optimization (2017, methodology): “…scenarios, including but not limited to methods for general regression and time-series models (McQuarrie and Tsai 1998), for regression when the sample size is small (Chapelle et al. 2002), for Bayesian topical trend analysis (Masada et al. 2009) and for log-linear models (Foo et al. 2007).”
Meta-Learning with Implicit Gradients (2019, background): “Such problems have been studied in the context of few-shot meta-learning (as discussed previously), gradient-based hyperparameter optimization [35, 46, 19, 11, 10], and a range of other settings [4, 31].”
Gradient-based Hyperparameter Optimization through Reversible Learning (2015, background): “(1998), Eigenmann & Nossek (1999), Chen & Hagan (1999), Bengio (2000), Abdel-Gawad & Ratner (2007), and Foo et al. (2008) showed that gradients of regularization parameters are available in closed form when training has converged exactly to a local minimum.”