The Split-Apply-Combine Strategy for Data Analysis

By Hadley Wickham. 2011

2,671 citations recorded by the source corpus.

Many data analysis problems involve the application of a split-apply-combine strategy, where you break up a big problem into manageable pieces, operate on each piece independently and then put all the pieces back together. This insight gives rise to a new R package that allows you to smoothly apply this strategy, without having to worry about the type of structure in which your data is stored. The paper includes two case studies showing how these insights make it easier to work with batting records for veteran baseball players and a large 3d array of spatio-temporal ozone measurements.

Subfields: Artificial Intelligence, Global and Planetary Change, Nature and Landscape Conservation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Global and Planetary Change

In Nature and Landscape Conservation

Citations in the corpus

Scientific Benchmarks for Guiding Macromolecular Energy Function Improvement (2013, methodology): “To do this, the feature analysis scripting framework uses the split-apply-combine strategy (Wickham, 2011): feature instances are grouped by their identifying columns, and for each group, a kernel density estimation (KDE) is computed over the measured columns.”
Comprehensiveness of conservation of useful wild plants: An operational indicator for biodiversity and sustainable development targets (2019, background): “…(Hunziker, 2017), PresenceAbsence (Freeman and Moisen, 2008), rJava (Urbanek, 2017), dismo (Hijmans et al., 2017), tidyverse (Wickham, 2017), SDMTools (van Der Wal et al., 2014), rgeos (Bivand and Rundel, 2017), shapefiles (Stabler, 2013), sp (Pebesma and Bivand, 2005), and plyr (Wickham, 2011).”
Cytofkit: A Bioconductor Package for an Integrated Mass Cytometry Data Analysis Pipeline (2016, methodology): “ClusterX addresses the memory issue with a splitapply-combine strategy [28], and automates density peak detection using generalized (extreme Studentized deviate) ESD test [29].”
Waste Not, Want Not: Why Rarefying Microbiome Data Is Inadmissible (2013, methodology): “These simulations, analyses, and graphics rely upon the cluster [55], foreach [56], ggplot2 [57], metagenomeSeq [40], phyloseq [32], plyr [58], re-shape2 [59], and ROCR [60] R packages; in addition to the DESeq(2) [13], edgeR [41], and PoiClaClu [48] R packages for RNA-Seq data, and tools available…”
How to handle missing data: A comparison of different approaches (2015, positive): “In the present study, we used the R-package (R Development Core Team, 2013) MICE (Multivariate Imputation by Chained Equations; Van Buuren & Groothuis-Oudshoorn, 2011; R package Plyr, Wickham, 2011; R package Zoo, Zeileis & Grothendieck, 2005).”
Two methods to find truth-value gaps and their application to the projection problem of homogeneity (2015, methodology): “(5)Graphs and statistics were created using the R programming language (R Core Team 2014) with the libraries ggplot2 (Wickham 2009), lme4 (Bates, Maechler, Bolker, and Walker 2014), reshape2 (Wickham 2007), and plyr (Wickham 2011).”
Handbook of Big Data (2016, methodology): “D&R is a statistical framework for data analysis based on the popular split-apply-combine paradigm [20].”
pdp: An R Package for Constructing Partial Dependence Plots (2017, neutral): “The partial function relies on the plyr package (Wickham, 2011), rather than R’s built-in for loops.”
Advancing the social identity theory of leadership: A meta-analytic review of leader group prototypicality (2020, methodology): “…software (version 3.6.1; R Core Team, 2016) within the RStudio environment (version 1.1.463; RStudio Team, 2016) using the packages Metafor (Viechtbauer, 2010), plyr (Wickham, 2011), and QuantPsyc (Fletcher, 2012), following guidelines provided by Viechtbauer (2016) and Assink and Wibbelink (2016).”
50 Years of Data Science (2017, background): “(cid:2) Two quick hits: For subfield “GDS 4: Data Visualization and Presentation,” one can mention several exemplary research contributions: Bill Cleveland’s work on statistical graphics (Cleveland et al. 1985; Cleveland 2013), along with Leland Wilkinson’s (Wilkinson 2006) and Hadley Wickham’s Wick”