The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, Out-of-Order Data Processing

By Tyler Akidau, Robert W. Bradshaw, Craig Chambers, Slava Chernyak, Rafael Fernández-Moctezuma, Reuven Lax, Sam McVeety, Daniel Mills, Frances Perry, E. Schmidt and 1 more. Proceedings of the VLDB Endowment, 2015

786 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Networks and Communications, Information Systems and Management

View Paper: https://www.semanticscholar.org/paper/fa63c3f53413ced7946623889c416e34a28676ea

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Information Systems and Management

Ion Stoica

Citations in the corpus

Online Machine Learning in Big Data Streams (2018, methodology): “Beam [12], based on the Google Cloud dataﬂow model [46, 11, 12], focuses on event time windows to process out-of-order data.”
Benchmarking Streaming Computation Engines: Storm, Flink and Spark Streaming (2016, background): “There are now several engines that are widely adopted, including Apache Storm [1], Apache Flink [2], Apache Spark (Spark Streaming) [3], [4], Apache Samza [5], Apache Apex [6] and Google Cloud Dataflow [7] among others.”
Live Video Analytics at Scale with Approximation and Delay-Tolerance (2017, background): “Stream processing systems used in production [2, 4, 62, 89] do not consider load-shedding, and resource-quality tradeoff and lag in their design; Google Cloud Dataflow [21] requires manual trade-off specifications.”
Cloud Programming Simplified: A Berkeley View on Serverless Computing (2019, background): “We note that many general purpose distributed frameworks (e.g., MapReduce, Apache Spark and Apache Beam/Cloud Dataflow [52]), parallel SQL engines (e.g., BigQuery, Azure Cosmos DB), as well as orchestration frameworks (e.g., Apache Airflow [53]) already produce such computation graphs internally.”
A Survey on Network Methodologies for Real-Time Analytics of Massive IoT Data and Open Research Issues (2017, background): “Apche Storm [96] [97] [98] [99] [100] Real-time stream processing, an open source distributed platform, resource aware scheduling Extremly low latency, massive scalability, fault tolerance, real-time distribution computation Micro-batch processing like Hadoop,Trident extension can increase delay, do”
A Repository of Conversational Datasets (2019, methodology): “Dataset creation scripts are written using Apache Beam and Google Cloud Dataﬂow (Akidau et al., 2015), which parallelizes the work across many machines.”
Understanding and Co-designing the Data Ingestion Pipeline for Industry-Scale RecSys Training (2021, methodology): “Our data ingestion pipeline performs ETL operations, similar to Apache Spark [49], Beam [9], and other query engines.”