Dryad: distributed data-parallel programs from sequential building blocks

By Michael Isard, Mihai Budiu, Yuan Yu, Andrew Birrell, Dennis Fetterly. 2007

766 citations recorded by the source corpus.

Subfields: Computer Networks and Communications, Hardware and Architecture, Information Systems

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Hardware and Architecture

In Information Systems

Citations in the corpus

TensorFlow: A system for large-scale machine learning (2016, positive): “We draw inspiration from the high-level programming models of dataflow systems [2, 21, 34] and the low-level efficiency of parameter servers [14, 20, 49].”
Pipe-SGD: A Decentralized Pipelined SGD Framework for Distributed Deep Net Training (2018, neutral): “To take advantage of parallelization across machines, a variety of approaches have been developed starting from the popular MapReduce paradigm [9, 51, 19, 37].”
A Survey on Distributed Machine Learning (2019, unknown): “Advanced variants have deliberated themselves from the strict tree topology of MapReduce data ows towards more exible structures like Forests (Dryad [75]) or generic Directed Acyclic Graphs (DAGs).”
Flexible Scheduling of Distributed Analytic Applications (2016, unknown): “Our algorithm is evaluated using trace-driven simulations, with large-scale real system traces: our flexible scheduler outperforms a baseline approach across a variety of metrics, including application turnaround times, and resource allocation efficiency.”
FlumeJava: easy, efficient data-parallel pipelines (2010, unknown): “MapReduce’s success in this domain inspired the development of a number of related systems, including Hadoop [2], LINQ/Dryad [20], and Pig [3].”
Piccolo: Building Fast, Distributed Programs with Partitioned Tables (2010, unknown): “Data-centric programming models [19, 27, 1], in which users are presented with a simplified interface to access data but no explicit communication mechanism, have proven a convenient and popular mechanism for expressing many computations.”
Log Clustering Based Problem Identification for Online Service Systems (2016, unknown): “In future, we will evaluate LogCluster on more projects on a variety of Cloud computing platforms such as Dryad [10].”
Large-scale Incremental Processing Using Distributed Transactions and Notifications (2010, unknown): “Batch processing systems like MapReduce [13, 22, 24] are well suited for efficiently transforming or analyzing an entire corpus: these systems can simultaneously use a large number of machines to process huge amounts of data quickly.”
Encoding, Fast and Slow: Low-Latency Video Processing Using Thousands of Tiny Threads (2017, unknown): “Apache Tez [24] and Dryad [13] support arbitrary DAG-structured computation, with data passed along edges in a computation graph.”
Splitwise: Efficient Generative LLM Inference Using Phase Splitting (2023, unknown): “Distributed dataflow systems orchestrate large-scale computational graphs and aim to provide general-purpose programmability [34], [46], [75], [82].”
Spark: Cluster Computing with Working Sets (2010, unknown): “MapReduce [11] pioneered this model, while systems like Dryad [16] and Map-Reduce- Merge [23] generalized the types of data flows supported.”
A Survey of Data-Intensive Scientific Workflow Management (2015, unknown): “Dryad [72] is another parallel processing framework developed by Microsoft (which eventually adopted Hadoop MapReduce).”
Swift: A language for distributed parallel scripting (2011, unknown): “Dryad [42] is an infrastructure for running data-parallel programs on a parallel or distributed system.”
An overview of the Hadoop/MapReduce/HBase framework and its current applications in bioinformatics (2010, unknown): “) Also, Indiana University (IU) researchers have performed comparisons [55] between MPI, Dryad (Microsoft [56,57]), Azure (Microsoft), and Hadoop MapReduce, measuring relative performance using three bioinformatics applications.”
HaLoop (2010, unknown): “MapReduce has been followed by a series of related systems including Dryad [10], Hive [9], Pig [14], and HadoopDB [2].”