An overview of bilevel optimization

By Benoît Colson, Patrice Marcotte, Gilles Savard. 2007

1,637 citations recorded by the source corpus.

Subfields: Computational Theory and Mathematics, Control and Systems Engineering, Numerical Analysis, Mathematical Optimization

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Theory and Mathematics

In Control and Systems Engineering

In Numerical Analysis

In Mathematical Optimization

Citations in the corpus

Using Machine Teaching to Identify Optimal Training-Set Attacks on Machine Learners (2015, methodology): “This equivalent single-level optimization problem allows us to use the descent method on an implicit function [9].”
Differentiable Neural Input Search for Recommender Systems (2020, background): “Equation 5 essentially defines a bi-level optimization problem [9], which has been studied in differentiable NAS [25] and gradient-based hyperparameter optimization [5, 13, 33].”
A Comprehensive Survey of Neural Architecture Search (2020, positive): “By solving a bilevel optimization problem [66, 67] to jointly optimize the probability of mixed operations (the parameters α of the network architecture) and network weightsw :”
Understanding and Robustifying Differentiable Architecture Search (2019, methodology): “We start by a short introduction of the bi-level optimization problem (Colson et al., 2007).”
Learning To Simulate (2018, background): “Attempting to solve it with a gradient-based approach poses multiple constraints on the lower-level problem 1b like smoothness, twice differentiability and an invertible Hessian (Bracken & McGill, 1973; Colson et al., 2007).”
Continual Learning of Control Primitives: Skill Discovery via Reset-Games (2020, background): “This form of sequential game is known as a Stackelberg game [49], and is closely connected to the field of bi-level optimization [10].”
Hierarchical Cooperative Multi-Agent Reinforcement Learning with Skill Discovery (2019, background): “Combining the learning at both levels, we view hierarchical MARL as a bilevel optimization problem [7]:”
Learning in Implicit Generative Models (2016, methodology): “Equation (5) allows us to specify a bi-level optimisation (Colson et al., 2007) by forming a ratio loss and a generative loss , using which we perform an alternating optimisation.”
Connecting Generative Adversarial Networks and Actor-Critic Methods (2016, background): “Bilevel optimization problems have been extensively studied in operations research, including the study of actor-critic methods [5], but largely in the setting where the two optimization problems are linear or convex programs [6].”
Deep Unfolding: Model-Based Inspiration of Novel Deep Architectures (2014, background): “In such cases, rigorous discriminative optimization of such models can be challenging because they may involve bi-level optimization, where the parameter optimization depends in turn on an iterative inference algorithm [3].”
Bilevel Programming for Hyperparameter Optimization and Meta-Learning (2018, positive): “In this paper, we consider bilevel optimization problems (see e.g. Colson et al., 2007) of the form where function f : Λ → R is deﬁned at λ ∈ Λ as We call E : R d × Λ → R the outer objective and, for every λ ∈ Λ , we call L λ : R d → R the inner objective .”
Stackelberg games for adversarial prediction problems (2011, background): “Numerous approaches that address bilevel programs have been studied, for instance, based on gradient descent, penalty function, and trust-region methods; see, for instance, [2] for a detailed survey.”
The Security of Latent Dirichlet Allocation (2015, methodology): “We formulate the optimal LDA attacking problem as a bilevel programming prob- Shike Mei, Xiaojin Zhu lem: (Bard 1998, Colson et al. 2007) The optimization for M in Eq (5) is called the upperlevel task, which is the attacker's optimization problem .”
AutoSpeech: Neural Architecture Search for Speaker Recognition (2020, neutral): “The NAS process can be viewed as a bi-level optimization problem [36] (eq.”
Self-Tuning Networks: Bilevel Optimization of Hyperparameters using Structured Best-Response Functions (2019, methodology): “Bilevel programs were first studied in economics to model leader/follower firm dynamics (Von Stackelberg, 2010) and have since found uses in various fields (see Colson et al. (2007) for an overview).”
iDARTS: Differentiable Architecture Search with Stochastic Implicit Gradients (2021, methodology): “The optimization in DARTS is based on the bi-level optimization formulation (Colson et al., 2007; Liu et al., 2019):”
Simplifying Architecture Search for Graph Neural Network (2020, methodology): “After designing the search space, denoted as A, the search process implies a bi-level optimization problem [5, 8], as show in the following:”
A Review on Bilevel Optimization: From Classical to Evolutionary Approaches and Applications (2017, background): “At this point, we would like to refer the readers to other review papers [49], [57], [90], [150], [164] and books [18], [60], [61], [140], [159] on bilevel optimization.”