Delay scheduling: a simple technique for achieving locality and fairness in cluster scheduling

By Matei A. Zaharia, Dhruba Borthakur, Joydeep Sen Sarma, Khaled Elmeleegy, Scott Shenker, Ion Stoica. European Conference on Computer Systems, 2010

1,562 citations recorded by the source corpus.

Subfields: Computer Networks and Communications, Information Systems

View Paper: https://www.semanticscholar.org/paper/860660d2cff0218b2628cc969ccbfbf5813482ee

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Information Systems

Citations in the corpus

Efficient coflow scheduling with Varys (2014, neutral): “It also differs from related problems like scheduling parallel tasks [9, 40] or caching parallel blocks [10]; unlike CPU or memory, the network involves coupled resources – each ﬂow’s progress depends on its rates at both source and destination.”
Spark: Cluster Computing with Working Sets (2010, methodology): “task to one of its preferred locations using a technique called delay scheduling [26].”
Shark: SQL and rich analytics at scale (2012, background): “For instance, Facebook and Microsoft have developed fair schedulers for Hadoop and Dryad that allow large historical queries, compute-intensive machine learning jobs, and short ad-hoc queries to safely coexist [32, 18].”
Toward Scalable Systems for Big Data Analytics: A Technology Tutorial (2014, background): “Many researchers are working to improve the scheduling policies in Hadoop, such as the delay scheduler [266], dynamic proportional scheduler [267], deadline constraint scheduler [268], and resource-aware scheduler [269].”
Apollo: Scalable and Coordinated Scheduling for Cloud-Scale Computing (2014, background): “Monolithic schedulers, such as Hadoop Fair Scheduler [28] and Quincy [16], implement a scheduling policy for an entire cluster using a centralized component.”
Resource Management with Deep Reinforcement Learning (2016, background): “We also currently do not take into account data locality considerations [41, 22].”
CIEL: A Universal Execution Engine for Distributed Data-Flow Computing (2011, background): “However, anecdotal evidence suggests that production Hadoop clusters mostly run jobs lasting less than 90 seconds [40].”
Sparrow: distributed, low latency scheduling (2013, methodology): “These policies mirror those offered by other schedulers, including the Hadoop Map Reduce scheduler [25].”
Jockey: guaranteed job latency in data parallel clusters (2012, background): “Finally, work-conserving allocation policies add variation by providing jobswith spare resources [12, 27].”
Multi-tenant GPU Clusters for Deep Learning Workloads: Analysis and Implications (2018, background): “In practice, similar to other scheduling regimes [8, 12, 27, 54], achieving both high locality and low waiting time is still challenging because these two requirements are often at odds with each other.”
Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing (2012, methodology): “Our scheduler assigns tasks to machines based on data locality using delay scheduling [32].”
Omega: flexible, scalable schedulers for large compute clusters (2013, background): “This two-level scheduling approach is used by a number of systems, including Mesos [13] and Hadoop-on-Demand (HOD) [4].”