Distributed computing in practice: the Condor experience

By D. Thain, Todd Tannenbaum, Miron Livny. Concurrency Practice and Experience, 2005

2,199 citations recorded by the source corpus.

Subfields: Computer Networks and Communications, Information Systems and Management

View Paper: https://www.semanticscholar.org/paper/0e6ecb5d48b05337ff67ae2d03a338f5afd9c429

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Information Systems and Management

Citations in the corpus

MapReduce (2020, neutral): “ Bulk Synchronous Programming [6]  MPI primitives [4]  Condor [5]  SAGA-MapReduce [8]  CGI-MapReduce [7]”
WorkflowSim: A toolkit for simulating scientific workflows in distributed environments (2012, neutral): “This overhead reflects the efficiency of the workflow scheduler (e.g., Condor [19]) to execute a job and the availability of resources for the execution of the job.”
Delay scheduling: a simple technique for achieving locality and fairness in cluster scheduling (2010, background): “Grids: Grid schedulers like Condor [28] support locality constraints, but usually at the level of geographic sites, because the jobs are more compute-intensive than MapReduce.”
Cloud Computing and Grid Computing 360-Degree Compared (2008, background): “At the fabric layer , Grids provide access to different resource types such as compute, storage and network resource, code repository, etc. Grids usually rely on existing fabric components, for instance, local resource managers (i.e. PBS [5], Condor [48], etc).”
Post-copy based live virtual machine migration using adaptive pre-paging and dynamic self-ballooning (2009, background): “Several distributed computing projects incorporate process migration (30; 23; 17; 29; 12; 6).”
Many-task computing for grids and supercomputers (2008, methodology): “High throughput computing [12], a term coined by Miron Livny within the Condor project [14], to contrast workloads for which the key metric is not floating per second (as in high performance computi month or year.”
PONDEROSA-C/S: client–server based software package for automated protein 3D structure determination (2014, methodology): “NMRFAM currently has a cluster of six servers, but we plan to expand the services by utilizing HTCondor (High-Throughput Condor; Thain et al. 2005) to enable nearly unlimited calculation resources.”
Apache Hadoop YARN: yet another resource negotiator (2013, background): “Prior to these recent efforts, there is a long history of work on resource management and scheduling - Condor [29], Torque [7], Moab [13] and Maui [20].”
Humanoid robots learning to walk faster: from the real world to simulation and back (2013, methodology): “The Condor workload management system [28] was used”
Dryad: distributed data-parallel programs from sequential building blocks (2007, neutral): “Condor [37] was an early example of such a system in a distributed setting, and we take more direct inspiration from three other models: shader languages developed for graphic processing units (GPUs) [30, 36], Google s MapReduce system [16], and parallel databases [18].”
Apollo: Scalable and Coordinated Scheduling for Cloud-Scale Computing (2014, background): “Job scheduling was extensively studied [24, 25] in high-performance computing for scheduling batch CPUintensive jobs and has become a hot topic again with emerging data-parallel systems [7, 15, 29].”
A Performance Analysis of EC2 Cloud Computing Services for Scientific Computing (2009, methodology): “Service type Examples VM,S Amazon (EC2 and S3), Mosso (+CloudFS), ... VM GoGrid, Joyent, infrastructures based on Condor Glide-in [28]/Globus VWS [12]/Eucalyptus [21], ...”
Serendipity: enabling remote computing among intermittently connected mobile devices (2012, methodology): “A major class of such jobs, supported by mainstream distributed computing platforms such as Condor [31], can be represented as a Directed Acyclic Graph (DAG).”
FireWorks: a dynamic workflow system designed for high‐throughput applications (2015, background): “However, we note that this matching is explicit and less ﬂ exible than more complex systems (e.g., as in Condor ’ s “ matchmaking ” service [18]).”