MillWheel: Fault-Tolerant Stream Processing at Internet Scale

By Tyler Akidau, Alex Balikov, K. Bekiroglu, Slava Chernyak, Josh Haberman, Reuven Lax, Sam McVeety, Daniel Mills, Paul Nordstrom, Paul Nordstrom and 1 more. 2013

645 citations recorded by the source corpus.

Subfields: Computer Networks and Communications, Information Systems, Information Systems and Management, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Information Systems

In Information Systems and Management

In ML Systems & Hardware

Citations in the corpus

Pathways: Asynchronous Distributed Dataflow for ML (2022, background): “…runtime must also support sparse data exchanges along sharded edges, in which messages can be sent between a dynamically chosen subset of shards, using standard progress tracking mechanisms (Akidau et al., 2013; Murray et al., 2013) to detect when all messages for a shard have been received.”
Discretized streams: fault-tolerant streaming computation at scale (2013, methodology): “window("5s") in the code above yields a D-Stream of RDDs containing the words in intervals [0,5), [1,6), [2,7), etc.”
Naiad: a timely dataflow system (2013, methodology): “Other tradeoffs may be more suitable for some applications: for example, MillWheel [5], a non-iterative streaming system with a programming model similar to the one de-ﬁned in § 2.2, writes per-key checkpoints for each batch of messages processed.”
Supporting Scalable Analytics with Latency Constraints (2015, methodology): “A number of systems have been developed to handle fast data, including Google MillWheel [2], Twitter Storm [37], Facebook Ptail This work is licensed under the Creative Commons AttributionNonCommercial-NoDerivs 3.0 Unported License.”
The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, Out-of-Order Data Processing (2015, methodology): “3.3.3 Billing: Triggers, Accumulation, & Retraction Two teams with billing pipelines built on MillWheel ex- perienced issues that motivated parts of the model.”
Online Machine Learning in Big Data Streams (2018, background): “advantage, stateful streaming systems have the possibility to emulate batch operation, hence the need for the lambda architecture can be eliminated. Beam [12], based on the Google Cloud dataﬂow model [46, 11, 12], focuses on event time windows to process out-of-order data. Beam can be connected to t”