Parsl: Pervasive Parallel Programming in Python
By Y. Babuji, A. Woodard, Zhuozhao Li, Daniel S. Katz, Ben Clifford, Rohan Kumar, Lukasz Lacinski, Ryan Chard, Justin M. Wozniak, Ian T Foster and 2 more. IEEE International Symposium on High-Performance Parallel Distributed Computing, 2019
327 citations recorded by the source corpus.
High-level programming languages such as Python are increasingly used to provide intuitive interfaces to libraries written in lower-level languages and for assembling applications from various components. This migration towards orchestration rather than implementation, coupled with the growing need for parallel computing (e.g., due to big data and the end of Moore's law), necessitates rethinking how parallelism is expressed in programs. Here, we present Parsl, a parallel scripting library that augments Python with simple, scalable, and flexible constructs for encoding parallelism. These constructs allow Parsl to construct a dynamic dependency graph of components that it can then execute efficiently on one or many processors. Parsl is designed for scalability, with an extensible set of executors tailored to different use cases, such as low-latency, high-throughput, or extreme-scale execution. We show, via experiments on the Blue Waters supercomputer, that Parsl executors can allow Python scripts to execute components with as little as 5 ms of overhead, scale to more than 250000 workers across more than 8000 nodes, and process upward of 1200 tasks per second. Other Parsl features simplify the construction and execution of composite programs by supporting elastic provisioning and scaling of infrastructure, fault-tolerant execution, and integrated wide-area data management. We show that these capabilities satisfy the needs of many-task, interactive, online, and machine learning applications in fields such as biology, cosmology, and materials science.
Subfields: Computer Networks and Communications, Information Systems, Information Systems and Management
View Paper: https://www.semanticscholar.org/paper/40ab4a3098e67e4e48deec94f0e3fe2932c51713
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Networks and Communications
In Information Systems
Citations in the corpus
- GenSLMs: Genome-scale language models reveal SARS-CoV-2 evolutionary dynamics (2022, background): “Training LLMs on SARS-CoV-2 genome datasets is non-trivial: (1) one needs to address the limitations of training LLMs with genomic sequence; and (2) one needs to overcome infrastructural challenges to enable LLM training on formidable sequence lengths in a reasonable time.”