Bayesian Optimization for Likelihood-Free Inference of Simulator-Based Statistical Models

By Michael U Gutmann, Jukka Corander. 2015

320 citations recorded by the source corpus.

Our paper deals with inferring simulator-based statistical models given some observed data. A simulator-based model is a parametrized mechanism which specifies how data are generated. It is thus also referred to as generative model. We assume that only a finite number of parameters are of interest and allow the generative process to be very general; it may be a noisy nonlinear dynamical system with an unrestricted number of hidden variables. This weak assumption is useful for devising realistic models but it renders statistical inference very difficult. The main challenge is the intractability of the likelihood function. Several likelihood-free inference methods have been proposed which share the basic idea of identifying the parameters by finding values for which the discrepancy between simulated and observed data is small. A major obstacle to using these methods is their computational cost. The cost is largely due to the need to repeatedly simulate data sets and the lack of knowledge about how the parameters affect the discrepancy. We propose a strategy which combines probabilistic modeling of the discrepancy with optimization to facilitate likelihood-free inference. The strategy is implemented using Bayesian optimization and is shown to accelerate the inference through a reduction in the number of required simulations by several orders of magnitude.

Subfields: Artificial Intelligence, Management Science and Operations Research, Statistical and Nonlinear Physics, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Statistical and Nonlinear Physics

In Probabilistic & Bayesian Methods

Citations in the corpus

Learning in Implicit Generative Models (2016, methodology): “, 2003), which has been successful for many problems with moderate dimension; and the natural choice of gradient-free optimisation methods, which include familiar tools such as Bayesian optimisation (Gutmann and Corander, 2016), evolutionary search like CMAES, and the Nelder-Mead method, amongst oth”
sbi: A toolkit for simulation-based inference (2020, methodology): “We are aware of several mature packages that implement SBI algorithms. elfi (Lintusaari et al., 2018) is a package offering BOLFI, a Gaussian process-based algorithm (Gutmann & Corander, 2016), and some classical ABC algorithms.”
Likelihood-free inference via classification (2014, methodology): “In the latter case, each simulated data set can immediately be compared toX(t0) which enables efficient iterative identification of parameter values with low discrepancy (Gutmann and Corander 2016).”
Fast Bayesian hyperparameter optimization on large datasets (2017, background): “Another interesting application would be to likelihood-free inference, where Bayesian optimization has been successfully applied before (Gutmann and Corander, 2016).”
Fast $\epsilon$-free Inference of Simulation Models with Bayesian Conditional Density Estimation (2016, methodology): “Bayesian optimization ABC [10] models p(‖x− xo‖ |θ) as a Gaussian process and then uses Bayesian optimization to guide simulations towards the region of small distances ‖x− xo‖.”
Likelihood-free MCMC with Amortized Approximate Ratio Estimators (2019, methodology): “by importance sampling. An important aspect of likelihood-free inference is minimizing the number of simulation calls while retaining the inference quality. Active simulation strategies such as BOLFI [52] and others [53, 54] achieve this by employing Bayesian optimization to make efﬁcient use of the”
Likelihood-Free Inference by Ratio Estimation (2016, methodology): “The models employed include regression models and neural networks, Gaussian processes as well as normalising ﬂows (Beaumont et al., 2002; Blum, 2010; Wilkinson, 2014; Gutmann and Corander, 2016; Papamakarios and Murray, 2016; Papamakarios et al., 2017, 2019; Chen and Gutmann, 2019).”
Adversarial Variational Optimization of Non-Differentiable Simulators (2017, unknown): “We compare our algorithm against ABC-SMC (Toni and Stumpf, 2009) and BOLFI (Gutmann and Corander, 2016) as baselines.”
Optimization Monte Carlo: Efficient and Embarrassingly Parallel Likelihood-Free Inference (2015, unknown): “Indeed, optimization as part of a likelihood-free inference procedure has recently been proposed [12]; using a probabilistic model of the mapping from parameters to differences between observations and simulator outputs, they apply “Bayesian Optimization” (e.”
Recurrent machines for likelihood-free inference (2018, methodology): “LFI methods (e.g., Beaumont et al., 2002; Gutmann and Corander, 2016; Louppe and Cranmer, 2017) are often based on handcrafted iterative optimization procedures, where a sequence of updates are performed to make the simulated data more similar to the observations.”
Likelihood-free MCMC with Approximate Likelihood Ratios (2019, methodology): “Active simulation strategies such as BOLFI [52] and others [53, 54] achieve this by employing Bayesian optimization to make efﬁcient use of the simulator.”
Probabilistic Symmetries and Invariant Neural Networks (2020, background): “…which predicts a distribution over labels; in implicit models like Generative Adversarial Networks (Goodfellow et al., 2014) or simulation-based models without a likelihood (e.g., Gutmann and Corander, 2016), and in many probabilistic programming languages (e.g., van de Meent et al., 2018), PY”
Automatic Posterior Transformation for Likelihood-Free Inference (2019, methodology): “On the M/G/1 model (Papamakarios & Murray, 2016), APT outperformed SNPE-A/B and approached SNL in performance (Fig.”
Under Review as a Conference Paper at Iclr 2017 Learning in Implicit Generative Models (2017, methodology): “…for many problems with moderate dimension; and the natural choice of gradientfree optimisation methods, which include familiar tools such as Bayesian optimisation (Gutmann and Corander, 2016), evolutionary search like CMA-ES, and the Nelder-Mead method, amongst others (Conn et al., 2009 ).”
Dynamic Likelihood-free Inference via Ratio Estimation (DIRE) (2018, methodology): “…discussed and proposed in (Dutta et al. 2016) and (Cranmer et al. 2015) can be used to amend or alleviate this shortcoming, e.g. via transfer learning and Bayesian optimization (Gutmann and Corander 2016), and the proposed method can seamlessly be combined with these kinds of improvements.”
Sequential Neural Likelihood: Fast Likelihood-free Inference with Autoregressive Flows (2018, methodology): “Bayesian Optimization Likelihood-free Inference [29] uses Bayesian optimization to ﬁnd parameter samples that minimize the distance x − x o .”
Likelihood-free inference with emulator networks (2018, methodology): “A recent proposal [14, 15] uses a GP to approximate the distribution of the discrepancy d ( s ( x ) , s ( x o )) as a function of θ (rather than the data x or summary statistics s ( x ) ), and uses ideas from Bayesian Optimization to propose new parameters.”