ARIA: automatic resource inference and allocation for mapreduce environments

By Abhishek Verma, L. Cherkasova, Roy H. Campbell. International Conference on Automation and Computing, 2011

504 citations recorded by the source corpus.

MapReduce and Hadoop represent an economically compelling alternative for efficient large scale data processing and advanced analytics in the enterprise. A key challenge in shared MapReduce clusters is the ability to automatically tailor and control resource allocations to different applications for achieving their performance goals. Currently, there is no job scheduler for MapReduce environments that given a job completion deadline, could allocate the appropriate amount of resources to the job so that it meets the required Service Level Objective (SLO). In this work, we propose a framework, called ARIA, to address this problem. It comprises of three inter-related components. First, for a production job that is routinely executed on a new dataset, we build a job profile that compactly summarizes critical performance characteristics of the underlying application during the map and reduce stages. Second, we design a MapReduce performance model, that for a given job (with a known profile) and its SLO (soft deadline), estimates the amount of resources required for job completion within the deadline. Finally, we implement a novel SLO-based scheduler in Hadoop that determines job ordering and the amount of resources to allocate for meeting the job deadlines. We validate our approach using a set of realistic applications. The new scheduler effectively meets the jobs' SLOs until the job demands exceed the cluster resources. The results of the extensive simulation study are validated through detailed experiments on a 66-node Hadoop cluster.

Subfields: Computer Networks and Communications, Information Systems

View Paper: https://www.semanticscholar.org/paper/2a7d3b967a356c2a42f729048b0d3511b0005351

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Information Systems

Citations in the corpus

Live Video Analytics at Scale with Approximation and Delay-Tolerance (2017, background): “Many systems [22, 39, 42, 56, 85] adaptively allocate resources to meet deadlines of batch jobs or reduce lag of streaming queries.”
Ernest: Efficient Performance Prediction for Large-Scale Advanced Analytics (2016, methodology): “These techniques have been aided by the inherent simplicity of the two-stage MapReduce model.”
CherryPick: Adaptively Unearthing the Best Cloud Configurations for Big Data Analytics (2017, background): “Our work on identifying the best cloud conﬁgurations is complementary to other works on identifying the best application conﬁgurations (e.g., [19, 38]).”