Heterogeneity and dynamicity of clouds at scale: Google trace analysis

By Charles Reiss, Alexey Tumanov, G. R. Ganger, Randy Howard Katz, Michael A. Kozuch. ACM Symposium on Cloud Computing, 2012

1,197 citations recorded by the source corpus.

Subfields: Computer Networks and Communications, Information Systems

View Paper: https://www.semanticscholar.org/paper/ca484d90aad294009ad09981b77517e7cd83b548

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Information Systems

Citations in the corpus

FIRM: An Intelligent Fine-Grained Resource Management Framework for SLO-Oriented Microservices (2020, positive): “, overprovisioning [36, 87], recurrent provisioning [54,66], and autoscaling [39,56,65,81,84, 88,127]) reduce SLO violations by allocating more CPUs and memory to microservice instances by using performance models, handcrafted heuristics (i.”
Autopilot: workload autoscaling at Google (2020, methodology): “One analysis [26] of a month-long trace of jobs executed at one of Google’s clusters [27] shows 50% average memory utilization; another analysis [23] of Alibaba’s YARN cluster showed tasks’ peak memory utilization never exceeding 80%.”
Effective Straggler Mitigation: Attack of the Clones (2013, neutral): “Analysis of these traces again reveal a heavytail distribution of job sizes, with 92% of the jobs accounting for only 2% of the overall resources [18].”
Quasar: resource-efficient and QoS-aware cluster management (2014, methodology): “Second, Quasar uses classification techniques to quickly and accurately determine the impact of the amount of resources (scale-out and scale-up), type of resources, and interference on performance for each workload and dataset.”
Task Failure Prediction in Cloud Data Centers Using Deep Learning (2019, methodology): “The Google cluster trace [25] starts at 19:00 EDT on Sunday May 1, 2011, and it records the resource utilization of CPU and memory usage of each task on the Google cluster of about 12.5k machines for 29 days.”
Borg: the next generation (2020, neutral): “We note that [27] claimed that Google compute consumption and memory consumption did not follow power-law distributions in 2011.”
Omega: flexible, scalable schedulers for large compute clusters (2013, methodology): “These results are broadly similar to other analyses of cluster traces from Yahoo [17], Facebook [7] and Google [20, 24, 25, 29].”
Efficient virtual memory for big memory servers (2013, background): “Second, the OS provisions physical memory for a primary region and maps all or part of the primary region through a direct segment by configuring the direct-segment registers.”
Addressing the straggler problem for iterative convergent parallel ML (2016, background): “Stragglers can occur for a number of reasons [5, 6], including hardware heterogeneity [32, 45, 49], hardware failures [6], unbalanced data distribution among tasks, garbage collection in high-level languages, and various OS effects [8, 42].”
Tachyon: Reliable, Memory Speed Storage for Cluster Computing Frameworks (2014, positive): “Our answer to this question is inﬂuenced by the following characteristics identiﬁed by previous works [20, 44] for data intensive applications: • Access Frequency : File access often follows a Zipf-like distribution (see [20, Figure 2]).”
Computer Science Review (neutral): “The most commonly used traces are the Google Cluster Trace [117] and the Alibaba Cluster Trace [118].”
Resource Management in Clouds: Survey and Research Challenges (2015, background): “[181] highlight the heterogeneity of both the resource types in the cluster and the manner in which they are used, pointing out that such heterogeneity reduces the effectiveness of traditional resource scheduling techniques.”
CPI2: CPU performance isolation for shared compute clusters (2013, background): “In one typical cluster, 7% of jobs run at production priority and use about 30% of the available CPUs, while non-production priority jobs consume about another 10% CPU [30].”
Who Limits the Resource Efficiency of My Datacenter: An Analysis of Alibaba Datacenter Traces (2019, background): “Early in 2012, [24] revealed that the average CPU and memory utilization in Google’s production cluster were only 20% and 40% respectively.”
Seer: Leveraging Big Data to Navigate the Complexity of Performance Debugging in Cloud Microservices (2019, background): “Cloud tracing & diagnostics: There is extensive related work on monitoring systems that has shown that execution traces can help diagnose performance, efficiency, and even security problems in large-scale systems [12, 24, 26, 42, 54, 77, 85, 93, 97].”
Resource Central: Understanding and Predicting Workloads for Improved Resource Management in Large Cloud Platforms (2017, background): “The closest prior works [8, 19, 22] did not address public cloud workloads.”