Veridical data science

By Bin Yu. 2019

217 citations recorded by the source corpus.

Significance Predictability, computability, and stability (PCS) are three core principles of data science. They embed the scientific principles of prediction and replication in data-driven decision making while recognizing the central role of computation. Based on these principles, we propose the PCS framework, including workflow and documentation (in R Markdown or Jupyter Notebook). The PCS framework aims at responsible, reliable, reproducible, and transparent analysis across fields of science, social science, engineering, business, and government. It can be used as a recommendation system for scientific hypothesis generation and experimental design. In particular, we propose (basic) PCS inference for reliability measures on data results, extending statistical inference to a much broader scope as current data science practice entails. Building and expanding on principles of statistics, machine learning, and scientific inquiry, we propose the predictability, computability, and stability (PCS) framework for veridical data science. Our framework, composed of both a workflow and documentation, aims to provide responsible, reliable, reproducible, and transparent results across the data science life cycle. The PCS workflow uses predictability as a reality check and considers the importance of computation in data collection/storage and algorithm design. It augments predictability and computability with an overarching stability principle. Stability expands on statistical uncertainty considerations to assess how human judgment calls impact data results through data and model/algorithm perturbations. As part of the PCS workflow, we develop PCS inference procedures, namely PCS perturbation intervals and PCS hypothesis testing, to investigate the stability of data results relative to problem formulation, data cleaning, modeling decisions, and interpretations. We illustrate PCS inference through neuroscience and genomics projects of our own and others. Moreover, we demonstrate its favorable performance over existing methods in terms of receiver operating characteristic (ROC) curves in high-dimensional, sparse linear model simulations, including a wide range of misspecified models. Finally, we propose PCS documentation based on R Markdown or Jupyter Notebook, with publicly available, reproducible codes and narratives to back up human choices made throughout an analysis. The PCS workflow and documentation are demonstrated in a genomics case study available on Zenodo.

Subfields: Artificial Intelligence, General Social Sciences

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In General Social Sciences

Citations in the corpus

SIRUS: making random forests interpretable (2019, methodology): “In that case, we argue that it is possible to deﬁne minimum requirements for interpretability through the triptych “simplicity, stability, and predictivity”, in line with the framework recently proposed by Yu and Kumbier (2019).”
Regression diagnostics meets forecast evaluation: conditional calibration, reliability diagrams, and coefficient of determination (2021, background): “…setting of identifiable functionals and real-valued outcomes, the arguments put forth by Dimitriadis et al. (2021) continue to apply, in that CORP estimators are bound to, simultaneously, improve statistical efficiency, reproducibility (Stodden et al., 2016), and stability (Yu and Kumbier, 2020).”
: "Statistical Modeling: The Two Cultures" (Statistical Science, 2001, 16(3), 199-231) (unknown): “Stability, reproducibility, and inference, require more than mere control of the prediction error (Yu, 2013; Yu and Kumbier, 2020).”
Bayesian Workflow (2020, unknown): “…and algorithmic, and this has placed workflow at the center of statistical practice (see, for example, Grolemund and Wickham, 2017, Bryan, 2017, and Yu and Kumbier, 2020), as well as in application areas (for example, Lee et al., 2019, discuss modeling workflow in psychology research).”
Improving the Interpretability of Neural Sentiment Classifiers via Data Augmentation (2019, background): “Moreover, the lack of interpretability on model prediction will raise the issue of trustworthy and fairness of sentiment classifiers in practice (Gilpin et al., 2018; Yu and Kumbier, 2019).”
Computational approaches streamlining drug discovery (2023, background): “One of the proposed concepts is the predictability, computability and stability framework for ‘veridical data science’ 90 .”
Interpretable Random Forests via Rule Extraction (2020, background): “…is no agreement in the machine learning litterature about a precise deﬁnition of interpretability (Lip-ton, 2016; Murdoch et al., 2019), it is yet possible to deﬁne simplicity, stability, and predictivity as minimum requirements for interpretable models (B ´ enard et al., 2019; Yu & Kumbier, 2019).”