Large-scale cluster management at Google with Borg
By Abhishek Verma, Luis Pedrosa, Madhukar R. Korupolu, David Oppenheimer, Eric Tune, John Wilkes. 2015
1,668 citations recorded by the source corpus.
Google's Borg system is a cluster manager that runs hundreds of thousands of jobs, from many thousands of different applications, across a number of clusters each with up to tens of thousands of machines. It achieves high utilization by combining admission control, efficient task-packing, over-commitment, and machine sharing with process-level performance isolation. It supports high-availability applications with runtime features that minimize fault-recovery time, and scheduling policies that reduce the probability of correlated failures. Borg simplifies life for its users by offering a declarative job specification language, name service integration, real-time job monitoring, and tools to analyze and simulate system behavior. We present a summary of the Borg system architecture and features, important design decisions, a quantitative analysis of some of its policy decisions, and a qualitative examination of lessons learned from a decade of operational experience with it.
Subfields: Computer Networks and Communications, Information Systems, ML Systems & Hardware
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Networks and Communications
- Ion Stoica
- Mosharaf Chowdhury
- Matei A. Zaharia
- Christina Delimitrou
- Christos Kozyrakis
- Hari Balakrishnan
- M. Franklin
- Raouf Boutaba
In Information Systems
- Ion Stoica
- Matei A. Zaharia
- Mosharaf Chowdhury
- Christina Delimitrou
- Christos Kozyrakis
- M. Franklin
- Shivaram Venkataraman
- Oriol Vinyals
In ML Systems & Hardware
- Ion Stoica
- Matei A. Zaharia
- M. Franklin
- Mosharaf Chowdhury
- Derek Gordon Murray
- Michael Isard
- Martín Abadi
- Xin Jin
Citations in the corpus
- TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems (2016, positive): “In our distributed environment, these different tasks are containers in jobs managed by a cluster scheduling system [51].”
- Flexible Scheduling of Distributed Analytic Applications (2016, neutral): “09 52 8v 4 Contents”
- Profiling a warehouse-scale computer (2015, background): “extensive tuning [47], the scheduler alone accounts for more than 5% of all datacenter cycles.”
- Optimus: an efficient dynamic resource scheduler for deep learning clusters (2018, methodology): “, Google uses Borg [63], Microsoft, Tencent and Baidu use YARN-like schedulers [61]) allocate a fixed amount of resources to each job upon its submission, according to resource requirements specified by the job owner.”
- Ernest: Efficient Performance Prediction for Large-Scale Advanced Analytics (2016, methodology): “This job submission API is similar to Hadoop’s Job API [10] and similar job submission APIs exist for dedicated clusters [65, 78] as well.”
- Multi-tenant GPU Clusters for Deep Learning Workloads: Analysis and Implications (2018, background): “Similar to clusters setup for big data analysis [13, 50], using shared clusters can facilitate better utilization and reduce development overheads.”
- Site Reliability Engineering: How Google Runs Production Systems (2016, methodology): “For more details on the similarities between Borg and Apache Mesos, see [Ver15].”
- Tiresias: A GPU Cluster Manager for Distributed Deep Learning (2019, background): “There are numerous existing resource managers and schedulers for CPU-based clusters for heterogenous workloads [20, 22–24, 26, 29, 38– 40, 46] or for traditional machine learning jobs [27, 37, 44].”
- Heterogeneity-Aware Cluster Scheduling Policies for Deep Learning Workloads (2020, background): “Traditional schedulers such as Mesos [34], Borg [59], TetriSched [56], and YARN [58] support workloads with fixed heterogeneous resource requests, but do not reason about the diverse performance characteristics of jobs across accelerators.”
- Shenango: Achieving High CPU Efficiency for Latency-sensitive Datacenter Workloads (2019, background): “In large-scale datacenters, even small improvements in CPU efficiency (the fraction of CPU cycles spent performing useful work) can save millions of dollars [72].”
- SAND: Towards High-Performance Serverless Computing (2018, background): “Similarly, Google has reported a median startup latency of 25 seconds on its internal container platform [55], 80% of which are attributed to library installation.”
- Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads (2019, background): “Similar to clusters setup for big data analysis [13, 48], using shared clusters can facilitate better utilization and reduce development overheads.”
- PipeSwitch: Fast Pipelined Context Switching for Deep Learning Applications (2020, background): “For example, Google Borg [1] packs online services and batch jobs, and saves 20%-30% machines (compared with not packing them).”