Towards a Rigorous Assessment of Systems Biology Models: The DREAM3 Challenges

By Robert J. Prill, Daniel Marbach, J. Sáez-Rodríguez, Peter K. Sorger, Leonidas G. Alexopoulos, Xiaowei Xue, Neil D. Clarke, G. Altan-Bonnet, Gustavo A. Stolovitzky. PLoS ONE, 2010

478 citations recorded by the source corpus.

Background Systems biology has embraced computational modeling in response to the quantitative nature and increasing scale of contemporary data sets. The onslaught of data is accelerating as molecular profiling technology evolves. The Dialogue for Reverse Engineering Assessments and Methods (DREAM) is a community effort to catalyze discussion about the design, application, and assessment of systems biology models through annual reverse-engineering challenges. Methodology and Principal Findings We describe our assessments of the four challenges associated with the third DREAM conference which came to be known as the DREAM3 challenges: signaling cascade identification, signaling response prediction, gene expression prediction, and the DREAM3 in silico network challenge. The challenges, based on anonymized data sets, tested participants in network inference and prediction of measurements. Forty teams submitted 413 predicted networks and measurement test sets. Overall, a handful of best-performer teams were identified, while a majority of teams made predictions that were equivalent to random. Counterintuitively, combining the predictions of multiple teams (including the weaker teams) can in some cases improve predictive power beyond that of any single method. Conclusions DREAM provides valuable feedback to practitioners of systems biology modeling. Lessons learned from the predictions of the community provide much-needed context for interpreting claims of efficacy of algorithms described in the scientific literature.

Subfields: Molecular Biology

View Paper: https://www.semanticscholar.org/paper/a595193f78e429286f6816e227a16b9753d835b3

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Molecular Biology

Citations in the corpus

Neural Granger Causality (2021, methodology): “Second, we compare our cMLP and cLSTM models with existing Granger causality approaches [33], [34] on the difﬁcult DREAM3 gene regulatory network recovery benchmark datasets [35] and ﬁnd that our methods outperform a wide set of competitors across all ﬁve datasets.”
Characterization and greedy learning of interventional Markov equivalence classes of directed acyclic graphs (2011, methodology): “3 DREAM4 Challenge We also measured the performance of GIES on synthetic gene expression data sets from the DREAM4 in silico challenge (Marbach et al., 2010; Prill et al., 2010).”
Evaluation of methods for modeling transcription-factor sequence specificity (2013, background): “The DREAM5 TF challenge 21-23 formed the original basis for the analyses presented here.”
Inferring Regulatory Networks from Expression Data Using Tree-Based Methods (2010, background): “DREAM4 Datasets The DREAM (for ‘‘Dialogue for Reverse Engineering Assessments and Methods’’) initiative organizes an annual reverse engineering competition called the DREAM challenge [34–37].”
Variational Causal Networks: Approximate Bayesian Inference over Causal Structures (2021, background): “Details of computing this is give in [30].”
Inference of complex biological networks: distinguishability issues and optimization-based solutions (2011, background): “Prill et al [27] further explored the issue of intrinsic impediments to network inference, designating identifiability of certain network edges and systematic false positives as the main barriers.”
Network Inference and Biological Dynamics (2011, background): “In particular, the Dialogue for Reverse Engineering Assessments and Methods (DREAM) challenges [Prill et al. (2010)] have provided an opportunity for objective empirical assessment of competing approaches.”
Learning linear cyclic causal models with latent variables (2012, methodology): “Data was then generated simulating a biologically plausible dynamical process and adding noise (Prill et al., 2010).”
ScholarWorks @ UTRGV ScholarWorks @ UTRGV (methodology): “After genes and SNPs were selected from related Genome-Wide Association Study (GWAS), it was tested how the method identify true positive edges between genes and SNPs without eQTL information.”
Ckmeans.1d.dp: Optimal k-means Clustering in One Dimension by Dynamic Programming (2011, methodology): “We applied our Ckmeans.1d.dp to quan-tize biological data in the DREAM Challenges (Prill et al., 2010) for qualitative dynamic modeling using generalized logical networks (Song et al., 2009).”
Comparison of co-expression measures: mutual information, correlation, and model based indices (2012, background): “For example, the DREAM (Dialogue for Reverse Engineering Assessments and Methods) project has repeatedly tackled this problem [41-43].”
A community computational challenge to predict the activity of pairs of compounds (2014, methodology): “( b ) Average and standard error over the 1,000 splits shown in a of the PC-index as computed in set S3 of individual teams (blue) and aggregates of the top-performing teams (red).”