Prediction uncertainty and optimal experimental design for learning dynamical systems.

By Benjamin Letham, P. A. Letham, Cynthia Rudin, E. Browne. 2015

21 citations recorded by the source corpus.

Dynamical systems are frequently used to model biological systems. When these models are fit to data, it is necessary to ascertain the uncertainty in the model fit. Here, we present prediction deviation, a metric of uncertainty that determines the extent to which observed data have constrained the model's predictions. This is accomplished by solving an optimization problem that searches for a pair of models that each provides a good fit for the observed data, yet has maximally different predictions. We develop a method for estimating a priori the impact that additional experiments would have on the prediction deviation, allowing the experimenter to design a set of experiments that would most reduce uncertainty. We use prediction deviation to assess uncertainty in a model of interferon-alpha inhibition of viral infection, and to select a sequence of experiments that reduces this uncertainty. Finally, we prove a theoretical result which shows that prediction deviation provides bounds on the trajectories of the underlying true model. These results show that prediction deviation is a meaningful metric of uncertainty that can be used for optimal experimental design.

Subfields: Artificial Intelligence, Genetics, Molecular Biology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Genetics

In Molecular Biology

Citations in the corpus

All Models are Wrong, but Many are Useful: Learning a Variable’s Importance by Studying an Entire Class of Prediction Models Simultaneously (2018, background): “This common scenario has been called the “Rashomon” effect of statistics (Breiman et al., 2001; see also Lecué, 2011; Statnikov et al., 2013; Tulabandhula and Rudin, 2014; Nevo and Ritov, 2017; Letham et al., 2016).”
All Models are Wrong but many are Useful: Variable Importance for Black-Box, Proprietary, or Misspecified Prediction Models, using Model Class Reliance (2018, background): “Letham et al. (2016) search for a pair of well-fitting dynamical systems models that give maximally different predictions.”
A study in Rashomon curves and volumes: A new perspective on generalization and model simplicity in machine learning (2019, methodology): “Rashomon sets: Rashomon sets have been used for various purposes (Breiman et al., 2001; Srebro et al., 2010; Fisher et al., 2019; Coker et al., 2018; Tulabandhula and Rudin, 2014b; Meinshausen and Bühlmann, 2010; Letham et al., 2016; Nevo and Ritov, 2017).”