FlumeJava: easy, efficient data-parallel pipelines

By Craig Chambers, Ashish Raniwala, Frances Perry, Stephen Adams, R. Henry, Robert W. Bradshaw, Nathan Weizenbaum. 2010

458 citations recorded by the source corpus.

Subfields: Computer Networks and Communications, Hardware and Architecture, Information Systems, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Hardware and Architecture

In Information Systems

In ML Systems & Hardware

Citations in the corpus

Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning (2018, methodology): “The Conceptual Captions dataset is programmat-ically created using a Flume (Chambers et al., 2010) pipeline.”
TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems (2016, neutral): “Its dataflow scheduler, which is the component that chooses the next node to execute, uses the same basic algorithm as Dryad, Flume, CIEL, and Spark.”
Boa: A language and infrastructure for analyzing ultra-large-scale software repositories (2013, background): “Last but not least, building analysis infrastructure to process ultra-largescale data efficiently can be very hard [8]–[10].”
Tupleware: Distributed Machine Learning on Small Clusters (2014, unknown): “Other programming models, such as FlumeJava [13], Ciel [27], and Piccolo [29] lack the low-level optimization potential that Tupleware’s algebra provides.”
Large-scale cluster management at Google with Borg (2015, methodology): “Many application frameworks have been built on top of Borg over the last few years, including our internal MapReduce system [23], FlumeJava [18], Millwheel [3], and Pregel [59].”
Large-Scale Visual Speech Recognition (2018, methodology): “Our pipeline makes heavy use of large-scale parallel processing and is implemented as a number of independent modules and filters on top of FlumeJava (Chambers et al., 2010).”
Scaling Graph Neural Networks with Approximate PageRank (2020, methodology): “For the first stage we use an efficient batch data processing pipeline [11] similar to MapReduce.”
Learning scheduling algorithms for data processing clusters (2018, background): “We focus on these jobs for two reasons: (i) many systems encode job stages and their dependencies as directed acyclic graphs (DAGs) [1, 3, 10, 22]; and (ii) scheduling DAGs is a hard algorithmic problem whose optimal solutions are intractable and difficult to capture in good heuristics [8].”
The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, Out-of-Order Data Processing (2015, methodology): “One particularly large log join pipeline runs in streaming mode on MillWheel by default, but has a separate FlumeJava batch implementation used for large scale backfills.”
Biperpedia: An Ontology for Search Applications (2014, methodology): “The pipeline is implemented as a FlumeJava pipeline [6].”
Dandelion: a compiler and runtime for heterogeneous systems (2013, methodology): “There are two main components: the Dandelion compiler generates the execution plans and the worker code to be run on the CPUs and GPUs of cluster machines, and the Dan-delion runtime uses the execution plans to manage the computation on the cluster, taking care of issues such as scheduling and…”
Online Machine Learning in Big Data Streams (2018, unknown): “Beam [12], based on the Google Cloud dataﬂow model [46, 11, 12], focuses on event time windows to process out-of-order data.”