FireWorks: a dynamic workflow system designed for high‐throughput applications

By Anubhav Jain, Shyue Ping Ong, Bharat K. Medasani, Xiaohui Qu, M. Kocher, Miriam Brafman, Guido Petretto, Gian-Marco Rignanese, Geoffroy Hautier, D. Gunter and 1 more. Concurrency and Computation, 2015

539 citations recorded by the source corpus.

This paper introduces FireWorks, a workflow software for running high‐throughput calculation workflows at supercomputing centers. FireWorks has been used to complete over 50 million CPU‐hours worth of computational chemistry and materials science calculations at the National Energy Research Supercomputing Center. It has been designed to serve the demanding high‐throughput computing needs of these applications, with extensive support for (i) concurrent execution through job packing, (ii) failure detection and correction, (iii) provenance and reporting for long‐running projects, (iv) automated duplicate detection, and (v) dynamic workflows (i.e., modifying the workflow graph during runtime). We have found that these features are highly relevant to enabling modern data‐driven and high‐throughput science applications, and we discuss our implementation strategy that rests on Python and NoSQL databases (MongoDB). Finally, we present performance data and limitations of our approach along with planned future work. Copyright © 2015 John Wiley & Sons, Ltd.

Subfields: Computer Networks and Communications, Information Systems and Management, Materials Chemistry

View Paper: https://www.semanticscholar.org/paper/2ae97d75f9f400118d376552095a8d99e799546e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Information Systems and Management

In Materials Chemistry

Citations in the corpus

Atomate: A high-level interface to generate, execute, and analyze computational materials science workflows (2017, methodology): “The FireWorks [28] library, which is a general workﬂow manager, as the language to deﬁne workﬂows and the mechanism to execute and manage workﬂows.”
Parsl: Pervasive Parallel Programming in Python (2019, methodology): “Systems like FireWorks support use cases that require concurrent execution of few ( < 1000) long-running tasks ( > 100 s) [33].”