Apache Hadoop YARN: yet another resource negotiator

By Vinod Kumar Vavilapalli, A. C. Murthy, C. Douglas, Sharad K. Agarwal, Mahadev Konar, R. Evans, Thomas Graves, J. Lowe, Hitesh Shah, S. Seth and 6 more. ACM Symposium on Cloud Computing, 2013

2,222 citations recorded by the source corpus.

The initial design of Apache Hadoop [1] was tightly focused on running massive, MapReduce jobs to process a web crawl. For increasingly diverse companies, Hadoop has become the data and computational agorá---the de facto place where data and computational resources are shared and accessed. This broad adoption and ubiquitous usage has stretched the initial design well beyond its intended target, exposing two key shortcomings: 1) tight coupling of a specific programming model with the resource management infrastructure, forcing developers to abuse the MapReduce programming model, and 2) centralized handling of jobs' control flow, which resulted in endless scalability concerns for the scheduler. In this paper, we summarize the design, development, and current state of deployment of the next generation of Hadoop's compute platform: YARN. The new architecture we introduced decouples the programming model from the resource management infrastructure, and delegates many scheduling functions (e.g., task fault-tolerance) to per-application components. We provide experimental evidence demonstrating the improvements we made, confirm improved efficiency by reporting the experience of running YARN on production environments (including 100% of Yahoo! grids), and confirm the flexibility claims by discussing the porting of several programming frameworks onto YARN viz. Dryad, Giraph, Hoya, Hadoop MapReduce, REEF, Spark, Storm, Tez.

Subfields: Computer Vision and Pattern Recognition, Information Systems, Information Systems and Management

View Paper: https://www.semanticscholar.org/paper/4a0bb4eece00f3e9445d1a0d933422aa408ce8d1

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Information Systems

In Information Systems and Management

Citations in the corpus

Computation Offloading Toward Edge Computing (2019, background): “Unlike the cloud that has the global view of overall resources and thus usually adopts centralized scheduling policies [115], [116], cloudlets are geographically distributed without a single centralized node, leading to difficulty in performing load balancing [117].”
Optimus: an efficient dynamic resource scheduler for deep learning clusters (2018, background): “Static resource allocation . jobs, e.g. , Mesos [40] in a TensorFlow cluster [23], Yarn [61] for the owner of a training job specifies resource requirements, e.g. ,”
Big data analytics on Apache Spark (2016, background): “Spark core runs over diverse clustermanagers including Hadoop YARN [76], ApacheMesos [39], Amazon EC2 and Spark’s built-in cluster manager (i.”
Resource Management with Deep Reinforcement Learning (2016, background): “This formulation is more practical than might appear since many cluster schedulers make independent scheduling decisions per machine (e.g., upon a hearbeat from the machine as in YARN [36]).”
Distributed data stream processing and edge computing: A survey on resource elasticity and future directions (2017, methodology): “ocessing element container that in turn delivers it to the appropriate PEs. 3.2.4. Apache Samza Apache Samza [60] is a stream processing framework that uses Apache Kafka for messaging and Apache YARN [61] for deployment, resource management, and security. A Samza application is a data ﬂow that consi”
Multi-tenant GPU Clusters for Deep Learning Workloads: Analysis and Implications (2018, methodology): “Given the resource fragmentation and the fact that the YARN scheduler is work-conserving, larger jobs could be additionally negatively affected by out-of-order scheduling.”
A survey on platforms for big data analytics (2014, background): “Hadoop YARN [8] is a resource management layer and schedules the jobs across the cluster.”
Occupy the cloud: distributed computing for the 99% (2017, methodology): “For example, PySpark, arguably one of the easier to use platforms, runs on top of Spark [49] (written in Scala) which interoperates and is closely coupled with HDFS [42] (written in Java), Yarn [46] (Java again), and the JVM.”
A cloud-scale acceleration architecture (2016, result): “HaaS manages FPGAs in a manner similar to Yarn [10] and other job schedulers.”
Learning scheduling algorithms for data processing clusters (2018, background): “In Decima, the feature vector (§5.1) xiv of a node v in job DAG i consists of: (i) the number of tasks 5We discuss Spark’s “standalone” mode of operation here (http://spark. apache.org/docs/latest/spark-standalone.html); YARN-based deployments can, in principle, use Decima, but require modifying bot”
Heterogeneity-Aware Cluster Scheduling Policies for Deep Learning Workloads (2020, background): “Traditional schedulers such as Mesos [34], Borg [59], TetriSched [56], and YARN [58] support workloads with ﬁxed heterogeneous resource requests, but do not reason about the diverse performance characteristics of jobs across accelerators.”