PACMan: Coordinated Memory Caching for Parallel Jobs

By Ganesh Ananthanarayanan, A. Ghodsi, A. Warfield, Dhruba Borthakur, Srikanth Kandula, Scott Shenker, Ion Stoica. Symposium on Networked Systems Design and Implementation, 2012

344 citations recorded by the source corpus.

View Paper: https://www.semanticscholar.org/paper/bb1d8795433bb221bdf75a31d644516474e53583

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Shark: SQL and rich analytics at scale (2012, background): “A study of Facebook’s Hive warehouse and Microsoft’s Bing analytics cluster showed that over 95% of queries in both systems could be served out of memory using just 64 GB/node as a cache, even though each system manages more than 100 PB of total data [6].”
Decentralized task-aware scheduling for data center networks (2014, background): “MapReduce [6] Overall, we ﬁnd that the distribution of task sizes depends on the application.”
Hopper: Decentralized Speculation-aware Cluster Scheduling at Scale (2015, background): “The scheduler places requests at the workers for its tasks; if a task has locality constraints, its requests are only placed on the workers meeting its constraints [13, 40, 49].”
Interactive Analytical Processing in Big Data Systems: A Cross-Industry Study of MapReduce Workloads (2012, background): “The workload is dominated by large-scale jobs with fixed computation patterns that are repeatedly and regularly run.”
Ernest: Efficient Performance Prediction for Large-Scale Advanced Analytics (2016, background): “Further, to handle cases where the number of tasks in a stage are greater than the number of cores available, we need adapt our estimate based on the number of waves [12] of tasks.”
Efficient Coflow Scheduling Without Prior Knowledge (2015, background): “Furthermore, it is a good indicator of actual size [41], because coflow size typically follows heavy-tailed distribution [20, 11].”
Making Sense of Performance in Data Analytics Frameworks (2015, methodology): “This has led to work on using the disk more efficiently [43] and caching data in memory [9] [30] [47] [51].”
Tachyon: Reliable, Memory Speed Storage for Cluster Computing Frameworks (2014, methodology): “For example, the same ﬁle, which represents a small dimension table in a data-warehouse, is repeatedly read by all mappers to do a map-side join with a fact table [12].”
Shark: fast data analysis using coarse-grained distributed memory (2012, methodology): “PACMan [9] takes advantage of coordinated caching at the filesystem level.”
Efficient coflow scheduling with Varys (2014, unknown): “It also differs from related problems like scheduling parallel tasks [9, 40] or caching parallel blocks [10]; unlike CPU or memory, the network involves coupled resources – each ﬂow’s progress depends on its rates at both source and destination.”
In-Memory Big Data Management and Processing: A Survey (2015, methodology): “There are general cache systems such as Memcached [61] and BigTable Cache [248], systems targeting speeding up analytics jobs such as PACMan [264] and Grid- Gain [51], and more purpose specific systems that have been designed for supporting specific frameworks such as NCache [265] for .”
Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics (2021, background): “Even without changing the data format, there are many types of caching strategies, auxiliary data structures and data layout strategies to explore for Lakehouses [4, 49, 53].”