Pyro: Deep Universal Probabilistic Programming

By Eli Bingham, Jonathan P. Chen, Martin Jankowiak, Fritz Obermeyer, Neeraj Pradhan, Theofanis Karaletsos, Rohit Singh, Paul A. Szerlip, Paul Horsfall, Noah D. Goodman. 2018

1,301 citations recorded by the source corpus.

Pyro is a probabilistic programming language built on Python as a platform for developing advanced probabilistic models in AI research. To scale to large datasets and high-dimensional models, Pyro uses stochastic variational inference algorithms and probability distributions built on top of PyTorch, a modern GPU-accelerated deep learning framework. To accommodate complex or model-specific algorithmic behavior, Pyro leverages Poutine, a library of composable building blocks for modifying the behavior of probabilistic programs.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Probabilistic & Bayesian Methods

Citations in the corpus

The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence (2020, background): “Uber's Pyro (Bingham et al., 2019) Arabshahi et al., (Arabshahi, Lu, Singh, & Anandkumar, 2019) show how a tree-LSTM can be augmented by an external memory that serves as a stack.”
Language Model Cascades (2022, methodology): “We implement cascades as a trace-based probabilistic programming language embedded in Python via effect handlers, inspired by Bingham et al. (2018); Phan et al. (2019), and via coroutines, inspired by Kochurov et al. (2019).”
Foundation Posteriors for Approximate Probabilistic Inference (2022, neutral): “Some require users to assist in the inference process by programmatically specifying conditional independencies [7] or hand-crafting variational distributions [50, 7, 17].”
Composable Effects for Flexible and Accelerated Probabilistic Programming in NumPyro (2019, methodology): “Within the Python community, some examples of embedded PPLs include Pyro [2] and ProbTorch [3] based on PyTorch [4], TensorFlow Probability [5] and Edward2 [6] based on TensorFlow, and PyMC3 [7] based on Theano.”
Importance of Search and Evaluation Strategies in Neural Dialogue Modeling (2018, methodology): “Inference We use no-u-turn (NUTS) sampler (Hoffman and Gelman, 2014) for posterior inference in Pyro (Bingham et al., 2018).”
Latency-Aware Neural Architecture Search with Multi-Objective Bayesian Optimization (2021, methodology): “To do inference in the SAAS model we rely on the open-source implementation of NUTS in Pyro (Bingham et al., 2019).”
Evaluation Examples are not Equally Informative: How should that change NLP Leaderboards? (2021, methodology): “All IRT models are implemented in Py-Torch (Paszke et al., 2019) and Pyro (Bingham et al., 2018).”
Bean Machine: A Declarative Probabilistic Programming Language For Efficient Programmable Inference (2020, background): “In contrast, other imperative Universal PPLs, such as Church (Goodman et al., 2008), Pyro (Bingham et al., 2019), Turing (Ge et al., 2018), and Gen (Cusumano-Towner et al., 2019), represent the statistical model as a monolithic program that explicitly draws a sample for each random variable that is ”
ProBO: Versatile Bayesian Optimization Using Any Probabilistic Programming Language (2019, methodology): “For PPLs using variational inference (VI), implicit models, or exact inference methods [50, 5, 14], inf computes the parameters of a distribution, and post draws a sample from this distribution.”
Inferring Rewards from Language in Context (2022, methodology): “We implement the full pragmatic model pL2(θ | u,M) in Pyro (Bingham et al., 2018) and use importance sampling to generate samples from the posterior over rewards.”
Meta-Learning an Inference Algorithm for Probabilistic Programs (2021, background): “, 2018) or defining proposal or variational distributions (Ritchie et al., 2015; Siddharth et al., 2017; Bingham et al., 2018; Tran et al., 2018; CusumanoTowner et al., 2019), all enabling the users to help inference or learning algorithms.”
Faster Attend-Infer-Repeat with Tractable Probabilistic Models (2019, methodology): “Finally, one could apply the example set by SuPAIR to deep probabilistic programming in general (Tran et al., 2017; Bingham et al., 2018), striving for a framework which combines generic probabilistic modeling with ameliorated inference, by using tractable models as subcomponents and inference machi”
Pathfinder: Parallel quasi-Newton variational inference (2021, neutral): “…standard black-box variational inference algorithm, implemented in Stan (Stan Development Team, 2021a), PyMC3 (Salvatier et al., 2016), Pyro (Bingham et al., 2019), TensorFlow Probability (Dillon et al., 2017), JAX (Bradbury et al., 2018), Turing.jl (Ge et al., 2018), and other…”
Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited (2020, methodology): “4 We then construct a Bayesian neural network with two hidden layers each with 20 units, no biases, and tanh activations, placing independent Gaussian (4)From the Bayesian neural network example in NumPyro (Phan et al., 2019; Bingham et al., 2019): https://github.”
Pareto Smoothed Importance Sampling (2015, methodology): “PSIS has been also implemented, for example, in the widely used Python packages ArviZ (Kumar et al., 2019) and Pyro (Bingham et al., 2019).”
MultiVerse: Causal Reasoning using Importance Sampling in Probabilistic Programming (2019, methodology): “First, it has been shown (Ness, 2019c,b) that in probabilistic programming languages, which support the intervention operation, such as Pyro (Bingham et al., 2018) (which has the do operator; or as it can be implemented in Edward (Tran et al.”
Deep Probabilistic Surrogate Networks for Universal Simulator Approximation (2019, background): “, 2016), and unrestricted (universal), which allow arbitrary models at the expense of making inference less efficient (Goodman et al., 2008; Mansinghka et al., 2014; Wood et al., 2014; Pfeffer, 2009; Goodman and Stuhlmüller, 2014; Bingham et al., 2018).”
Comparing Test Sets with Item Response Theory (2021, methodology): “We use variational inference to infer IRT parameters from model response patterns using Pyro (Ran-ganath et al., 2014; Bingham et al., 2019).”