Quincy: fair scheduling for distributed computing clusters

By Michael Isard, Vijayan Prabhakaran, J. Currey, Udi Wieder, Kunal Talwar, Andrew Vladislav Goldberg. Symposium on Operating Systems Principles, 2009

1,015 citations recorded by the source corpus.

Subfields: Computer Networks and Communications, Information Systems

View Paper: https://www.semanticscholar.org/paper/da411a75ffc80d1101c1c1ce21a966e27eec1aec

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Information Systems

Citations in the corpus

Large-scale cluster management at Google with Borg (2015, methodology): “Quincy [49] uses a network ﬂow model to provide fairness-and data locality-aware scheduling for data-processing DAGs on clusters of a few hundred nodes.”
Flexible Scheduling of Distributed Analytic Applications (2016, background): “Similarly, Quincy [17] and DRF [43] study maxmin fair, task-level resource allocation, specifically working on multi-dimensional resources.”
Sharing the Data Center Network (2011, background): “Many pieces of today’s datacenter ecosystem use topology information (e.g., Map-Reduce schedulers [27] and VM placement algorithms).”
PARTIES: QoS-Aware Resource Partitioning for Multiple Interactive Services (2019, background): “First, there are cluster schedulers that infer the expected interference of a given application colocation [13, 18, 20–22, 24, 33, 45, 58, 59, 61, 62], and either adjust allocations at runtime, or completely disallow resource sharing when the predicted latency violates QoS.”
Jockey: guaranteed job latency in data parallel clusters (2012, methodology): “A compiler translates the job into an execution plan graph wherein nodes represent stages such as map, reduce or join, and edges represent dataow [7, 9, 12].”
Occupy the cloud: distributed computing for the 99% (2017, background): “Scalable Scheduling: A number of cluster scheduling papers [21, 32, 33, 40] have looked at providing low latency scheduling for data parallel frameworks running on servers.”
Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads (2019, background): “The effect of locality constraints on queuing delays has been extensively explored in large-scale resource allocation [8,12,27,52].”
Themis: Fair and Efficient GPU Cluster Scheduling for Machine Learning Workloads (2019, background): “, Quincy [15], DRF [8], and Carbyne [10].”
ARIA: automatic resource inference and allocation for mapreduce environments (2011, background): “Similar fairness and data locality goals are pursued in Quincy scheduler [8] proposed for the Dryad environment [7].”
Gandiva: Introspective Cluster Scheduling for Deep Learning (2018, background): “Most recent big data scheduling frameworks assume jobs are modeled after a data flow graph (DFG) [26, 55, 13, 27, 20, 22].”
Twine: A Unified Cluster Management System for Shared Infrastructure (2020, background): “A large body of work [6, 15, 20, 31] focuses on improving batch scheduling throughput and latency.”
Multi-resource packing for cluster schedulers (2014, background): “(ii) Schedulers also ignore disk and network requirements of tasks [5, 12, 18].”
Reining in the Outliers in Map-Reduce Clusters using Mantri (2010, methodology): “Following on the map-reduce [10] paper, there has been work in improving workflows [1, 12], language design [7, 21, 31], fair schedulers [13, 32], and providing privacy [25].”