Pegasus: A framework for mapping complex scientific workflows onto distributed systems

By Ewa Deelman, Gurmeet Singh, Mei-Hui Su, James Blythe, Yolanda Gil, Carl Kesselman, Gaurang Mehta, Karan Vahi, G. Berriman, John Good and 3 more. Scientific Programming, 2005

1,448 citations recorded by the source corpus.

This paper describes the Pegasus framework that can be used to map complex scientific workflows onto distributed resources. Pegasus enables users to represent the workflows at an abstract level without needing to worry about the particulars of the target execution systems. The paper describes general issues in mapping applications and the functionality of Pegasus. We present the results of improving application performance through workflow restructuring which clusters multiple tasks in a workflow into single entities. A real-life astronomy application is used as the basis for the study.

Subfields: Computer Networks and Communications, Information Systems and Management

View Paper: https://www.semanticscholar.org/paper/2022160699559f695d4007e6328cc01261d3beed

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Information Systems and Management

Citations in the corpus

Cost optimized provisioning of elastic resources for application workflows (2011, methodology): “First, a user asks the Workflow Management System (WMS) to execute a workflow within a deadline for a given resource specification.”
Evolutionary Multi-Objective Workflow Scheduling in Cloud (2016, neutral): “F”
List Scheduling Algorithm for Heterogeneous Systems by an Optimistic Cost Table (2014, methodology): “5, 1, 2] for heterogeneity and [2, 4, 8, 16, 32] for CPU number.”
DataHub: Collaborative Data Science & Dataset Version Management at Scale (2014, methodology): “Most data science systems, including analytic packages like SAS, Excel, R, Matlab, Mahout, Scikit [22], and Pandas [21]; workflow tools like Pegasus [4], Chimera [6], and VisTrails [1]; and collaboration tools like Fusion tables [12], Orchestra [8], and CQMS [10, 7] lack dataset versioning managemen”
Swift: A language for distributed parallel scripting (2011, background): “Pegasus [41] is primarily a set of DAG transformers.”
Cost- and deadline-constrained provisioning for scientific workflow ensembles in IaaS clouds (2012, methodology): “Our experience suggests that such assumption is often reasonable, since we can obtain workflow performance characteristics from preliminary runs [22], [25], [26].”
Scheduling Algorithms for Grid Computing: State of the Art and Open Problems (2006, background): “When there are multiple appropriate resources available, CWG supports a few standard selection algorithms: random, round-robin and min-min [38].”
Cloud auto-scaling with deadline and budget constraints (2010, background): “They are just-in-time scheduling (fully dynamic approach) [43], full-ahead planning (fully static approach) [44][45] and hybrid scheduling [46][47].”
A CyberGIS Framework for the Synthesis of Cyberinfrastructure, GIS, and Spatial Analysis (2010, methodology): “Workﬂow and provenance services manage the interactions between domain decomposition and task scheduling and support ﬂexible composition and efﬁcient execution of GIS and spatial analysis work-ﬂows based on user-centric management of computation jobs (Deelman et al. 2005; Walker et al. 2007).”
FireWorks: a dynamic workflow system designed for high‐throughput applications (2015, background): “A ﬁ rst comparison can be made to the Pegasus [12] and Condor DAGMan [34] systems.”
Falkon: a Fast and Light-weight tasK executiON framework (2007, methodology): “have also created a task-graph implementation of the Montage code, using Pegasus and DAGman [33].”
A Particle Swarm Optimization-Based Heuristic for Scheduling Workflow Applications in Cloud Computing Environments (2010, background): “They have proposed Pegasus [5], which is a framework that maps complex scientific workflows onto distributed resources such as the Grid.”
Flexible IO and integration for scientific codes through the adaptable IO system (ADIOS) (2008, background): “ACM acknowledges that this contribution was authored or co-authored by an employee, contractor or af.liate of the U.S. Government.”
Many-task computing for grids and supercomputers (2008, background): “This application is both compute intensive and data intensive, and has been run as both a HTC (using Pegasus/DAGMan [8], and Condor [14]) and a HPC (using MPI) application, but we found its scalability to be limited when run under HTC or HPC.”
G-Hadoop: MapReduce across distributed data centers for data-intensive computing (2013, methodology): “Currently data-intensive workflow systems, such as DAGMan [6], Pegasus [7], Swift [8], Kepler [9], Virtual Workflow [10,11], Virtual Data System [12] and Taverna [13], are used for distributed data processing across multiple data centers.”
ProvDB: A System for Lifecycle Management of Collaborative Analysis Workflows (2016, background): “Pegasus: A framework for mapping complex scientific workflows onto distributed systems.”