Sustainable data analysis with Snakemake
By Felix Mölder, K. Jablonski, B. Letcher, M. Hall, C. Tomkins-Tinch, Vanessa V. Sochat, J. Forster, Soohyun Lee, Sven Twardziok, A. Kanitz and 5 more. F1000Research, 2021
1,549 citations recorded by the source corpus.
Data analysis often entails a multitude of heterogeneous steps, from the application of various command line tools to the usage of scripting languages like R or Python for the generation of plots and tables. It is widely recognized that data analyses should ideally be conducted in a reproducible way. Reproducibility enables technical validation and regeneration of results on the original or even new data. However, reproducibility alone is by no means sufficient to deliver an analysis that is of lasting impact (i.e., sustainable) for the field, or even just one research group. We postulate that it is equally important to ensure adaptability and transparency. The former describes the ability to modify the analysis to answer extended or slightly different research questions. The latter describes the ability to understand the analysis in order to judge whether it is not only technically, but methodologically valid. Here, we analyze the properties needed for data analysis to become reproducible, adaptable, and transparent. We show how the popular workflow management system Snakemake can be used to guarantee this, and how it enables an ergonomic, combined, unified representation of all steps involved in data analysis, ranging from raw data processing, to quality control and fine-grained, interactive exploration and plotting of final results.
View Paper: https://www.semanticscholar.org/paper/bff639f7b02be17dca93ee0d8515144fbd79944e
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
- Mosè Manni
- Matthew Berkeley
- Mathieu Seppey
- Max Schubach
- E. Zdobnov
- Thorben Maass
- Lusiné Nazaretyan
- Sebastian Röner
Citations in the corpus
- RNA modifications detection by comparative Nanopore direct RNA sequencing (2019, unknown): “qRT-PCR primers: 7sk (22-73): Fwd 5′-GCGACATCTGTCACCC CATT-3′; Rev 5′-CAGCCAGATCAGCCGAATCA-3′.”
- BUSCO Update: Novel and Streamlined Workflows along with Broader and Deeper Phylogenetic Coverage for Scoring of Eukaryotic, Prokaryotic, and Viral Genomes (2021, methodology): “We provide an example using Snakemake (Mölder et al. 2021) at https://gitlab.”
- The Galaxy platform for accessible, reproducible, and collaborative data analyses: 2024 update (2024, unknown): “Additionally, a ‘Data science’ topic has been added covering basics of Python, R, SQL and snakemake ( 19 ) using interactive notebooks which can be run inside Galaxy as interactive tools, or independently of Galaxy.”
- CADD v1.7: using protein language models, regulatory CNNs and other nucleotide-level scores to improve genome-wide variant predictions (2024, methodology): “Offline scoring is based on the workflow management sys-tem Snakemake ( 75 ) with dependency management through conda ( https://conda.io ).”
- BUSCO: Assessing Genomic Data Quality and Beyond (2021, background): “…Project (i5K Consortium, 2013), the 10,000 Plant Genomes Project (10KP; Cheng et al., 2018), the GEM (Genomes from Earth’s Microbiomes) catalog (Nayfach et al., 2021), and the European Reference Genome Atlas (ERGA) aim at sequencing hundreds of thousands of new genomes in the coming years…”