CIEL: A Universal Execution Engine for Distributed Data-Flow Computing

By Derek Gordon Murray, Malte Schwarzkopf, Christopher Smowton, Steven Smith, Anil Madhavapeddy, S. Hand. 2011

335 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In ML Systems & Hardware

Citations in the corpus

Toward Scalable Systems for Big Data Analytics: A Technology Tutorial (2014, background): “In addition, other research has focused on programming models for more specific tasks, such as cross joining two sets [173], iterative computation [174], [175], in-memory computation with fault-tolerance [176], incremental computation [177]–[180], and data-dependent flow control decision [181].”
Tupleware: Distributed Machine Learning on Small Clusters (2014, critical): “Other programming models, such as FlumeJava [13], Ciel [27], and Piccolo [29] lack the low-level optimization potential that Tupleware’s algebra provides.”
A Berkeley View of Systems Challenges for AI (2017, background): “Data parallel systems [55, 79, 114] handle orders of magnitude fewer tasks per sec, while HPC and distributed DL systems [2, 23, 82]”
Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center (2011, background): “Prominent examples include MapReduce [18], Dryad [24], MapReduce Online [17] (which supports streaming jobs), Pregel [28] (a specialized framework for graph computations), and others [27, 19, 30].”
Coded TeraSort (2017, background): “Speciﬁcally, distributed systems like Apache Spark [1] and computational primitives like MapReduce [2], Dryad [3], and CIEL [4] have gained signiﬁcant traction, as they enable the execution of production-scale tasks on data sizes of the order of tens of terabytes and more.”
Faasm: Lightweight Isolation for Efficient Stateful Serverless Computing (2020, background): “Although focuses on ﬁxed-size clusters and not ﬁne-grained elastic scaling or multi-tenancy, distributed dataﬂow systems such as Naiad [50], SDGs [25] and CIEL [51] provide high-level interfaces for distributed state, with similar aims to those of distributed data objects—they could be implemented i”
Jockey: guaranteed job latency in data parallel clusters (2012, background): “Hence, we believe Jockey is a better match for production DAG-like frameworks such as Hive [22], Pig [17], and Ciel [16] To predict the completion time of a running job, Jockey must estimate the job s current progress.”
MillWheel: Fault-Tolerant Stream Processing at Internet Scale (2013, background): “Ciel [25] targets general data processing problems, while dynamically generating the dataflow graph.”
Big Data: A Survey (2014, background): “, iterative computations [105, 106], fault-tolerant memory computations [107], incremental computations [108], and flow control decisionmaking related to data [109].”