Efficient coflow scheduling with Varys

By Mosharaf Chowdhury, Yinmin Zhong, Ion Stoica. Conference on Applications, Technologies, Architectures, and Protocols for Computer Communication, 2014

484 citations recorded by the source corpus.

Communication in data-parallel applications often involves a collection of parallel flows. Traditional techniques to optimize flow-level metrics do not perform well in optimizing such collections, because the network is largely agnostic to application-level requirements. The recently proposed coflow abstraction bridges this gap and creates new opportunities for network scheduling. In this paper, we address inter-coflow scheduling for two different objectives: decreasing communication time of data-intensive jobs and guaranteeing predictable communication time. We introduce the concurrent open shop scheduling with coupled resources problem, analyze its complexity, and propose effective heuristics to optimize either objective. We present Varys, a system that enables data-intensive frameworks to use coflows and the proposed algorithms while maintaining high network utilization and guaranteeing starvation freedom. EC2 deployments and trace-driven simulations show that communication stages complete up to 3.16X faster on average and up to 2X more coflows meet their deadlines using Varys in comparison to per-flow mechanisms. Moreover, Varys outperforms non-preemptive coflow schedulers by more than 5X.

Subfields: Computer Networks and Communications, Hardware and Architecture, Information Systems

View Paper: https://www.semanticscholar.org/paper/231ba17921ebd80e95771e28dfb5082e169d5a53

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

1. Ion Stoica
2. Mosharaf Chowdhury
3. Scott Shenker
4. Jennifer Rexford
5. Sylvia Ratnasamy
6. Changhoon Kim
7. Chuanxiong Guo
8. Anirudh Sivaraman

In Hardware and Architecture

1. Mosharaf Chowdhury
2. Ion Stoica
3. Scott Shenker
4. Chuanxiong Guo
5. Jennifer Rexford
6. Anirudh Sivaraman

In Information Systems

1. Mosharaf Chowdhury
2. Ion Stoica
3. Scott Shenker
4. Kay Ousterhout
5. Anirudh Sivaraman
6. Jennifer Rexford
7. Chuanxiong Guo
8. Yibo Zhu

Citations in the corpus

• Efficient Coflow Scheduling Without Prior Knowledge (2015, methodology): “first (as in Varys [20]) for heavy-tailed coflows, the proposed scheduler works well in practice.”
• HULA: Scalable Load Balancing Using Programmable Data Planes (2016, methodology): “The most commonly used dataplane load balancing technique is equal-cost multi-path routing (ECMP), which spreads traffic by assigning each flow to one of several paths at random.”
• Making Sense of Performance in Data Analytics Frameworks (2015, background): “Researchers have used the argument that data-intensive application performance is closely tied to datacenter network performance to justify a wide variety of network optimizations [6,14,17–21,27,28,41,42,48,53].”
• A generic communication scheduler for distributed DNN training acceleration (2019, methodology): “, Stale Synchronous Parallel [20] for PS and different all-reduce algorithms [10, 14]; (4) minimizing network flow completion time by using flow control [26] or Coflow scheduling [13].”