Dask: Parallel Computation with Blocked algorithms and Task Scheduling

By Matthew Rocklin. SciPy, 2015

783 citations recorded by the source corpus.

Dask enables parallel and out-of-core computation. We couple blocked algorithms with dynamic and memory aware task scheduling to achieve a parallel and out-of-core NumPy clone. We show how this extends the effective scale of modern hardware to larger datasets and discuss how these ideas can be more broadly applied to other parallel collections.

Subfields: Computer Networks and Communications, Hardware and Architecture, Information Systems

View Paper: https://www.semanticscholar.org/paper/73b58192f30bb6be8e798084d4481b97124570ed

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Hardware and Architecture

In Information Systems

Citations in the corpus

Mobile phone location data for disasters: A review from natural hazards and epidemics (2021, positive): “To enable non-experts to use the soft-wares, the codes are optimized using Dask(Rocklin, 2015) for parallel computing, so that analysis on massive mobility datasets can be conducted under constrained resources, on local laptop or desktop computers.”
SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization (2021, methodology): “Furthermore, SMAC3 supports two kinds of parallelization techniques: (i) via DASK (Rocklin, 2015) and (ii) via running an arbitrary number of independent SMAC3 instances exchanging information via the ﬁle system.”
CellProfiler 4: improvements in speed, utility and usability (2021, background): “The large file sizes associated with tissue specimens pose a challenge for image analysis, as system memory typically is not sufficient to load the entire image at once, a bottleneck which could be avoided by adopting packages such as Dask [13] as a means of handling such images by loading subsectio”
Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges (2021, methodology): “, 2022) also allows for multi-objective optimization, for parallel optimization with Dask (Rocklin, 2015) and for specification of expert priors to make use of existing human expertise (Hvarfner et al.”
Optimizing data-intensive computations in existing libraries with split annotations (2018, methodology): “In recognition of this bottleneck, researchers have proposed redesigning software libraries to use optimizing compilers and runtimes [27, 46, 48, 56, 61, 63–65].”
The SunPy Project: Open Source Development and Status of the Version 1.0 Core Package (2020, background): “…These core packages form the backbone of hundreds of additional scientific Python packages, such as astropy for functionality and tools specific to astronomy, scikit-learn for machine learning and data mining (Pedregosa et al. 2011), and dask for parallel and distributed computing (Rocklin 2015).”