ProBO: a Framework for Using Probabilistic Programming in Bayesian Optimization

By Willie Neiswanger, Kirthevasan Kandasamy, Barnab'as P'oczos, Jeff Schneider, Eric P. Xing. 2019

18 citations recorded by the source corpus.

Optimizing an expensive-to-query function is a common task in science and engineering, where it is beneficial to keep the number of queries to a minimum. A popular strategy is Bayesian optimization (BO), which leverages probabilistic models for this task. Most BO today uses Gaussian processes (GPs), or a few other surrogate models. However, there is a broad set of Bayesian modeling techniques that we may want to use to capture complex systems and reduce the number of queries. Probabilistic programs (PPs) are modern tools that allow for flexible model composition, incorporation of prior information, and automatic inference. In this paper, we develop ProBO, a framework for BO using only standard operations common to most PPs. This allows a user to drop in an arbitrary PP implementation and use it directly in BO. To do this, we describe black box versions of popular acquisition functions that can be used in our framework automatically, without model-specific derivation, and show how to optimize these functions. We also introduce a model, which we term the Bayesian Product of Experts, that integrates into ProBO and can be used to combine information from multiple models implemented with different PPs. We show empirical results using multiple PP implementations, and compare against standard BO methods.

Subfields: Artificial Intelligence, Computational Theory and Mathematics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Sam Nolen
2. Colin White
3. Yash Savani
4. Willie Neiswanger

In Computational Theory and Mathematics

1. Yash Savani
2. Colin White
3. Sam Nolen
4. Willie Neiswanger

Citations in the corpus

• BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search (2019, methodology): “Following previous work (Neiswanger et al. 2019), we use the following acquisition function estimates for an input architecture a ∈ A: φTS(x) = fm̃(x), m̃ ∼ Unif (1,M) (1) φITS(x) = f̃x(x), f̃x(x) ∼ N (f̂ , σ̂2) (2) φUCB(x) = f̂ − βσ̂ (3) φEI(a) = E [1 [fm(a) > ymin] (ymin − fm(a))] (4) = ∫ ymin −∞…”
• BoTorch: Programmable Bayesian Optimization in PyTorch (2019, unknown): “ProBO (Neiswanger et al., 2019) and GPFlowOpt (Knudde et al.”
• Local Search is State of the Art for Neural Architecture Search Benchmarks (2020, methodology): “For vanilla Bayesian optimization, we used the ProBO implementation [24].”
• A Study on Encodings for Neural Architecture Search (2020, methodology): “We use the ProBO [12] implementation, which uses a Gaussian process kernel and expected improvement as the acquisition function.”