High-availability algorithms for distributed stream processing

By Jeong-Hyon Hwang, M. Balazinska, A. Rasin, Ugur Çetintemel, M. Stonebraker, S. Zdonik. IEEE International Conference on Data Engineering, 2005

324 citations recorded by the source corpus.

Stream-processing systems are designed to support an emerging class of applications that require sophisticated and timely processing of high-volume data streams, often originating in distributed environments. Unlike traditional data-processing applications that require precise recovery for correctness, many stream-processing applications can tolerate and benefit from weaker recovery guarantees. In this paper, we study various recovery guarantees and pertinent recovery techniques that can meet the correctness and performance requirements of stream-processing applications. We discuss the design and algorithmic challenges associated with the proposed recovery techniques and describe how each can provide different guarantees with proper combinations of redundant processing, checkpointing, and remote logging. Using analysis and simulations, we quantify the cost of our recovery guarantees and examine the performance and applicability of the recovery techniques. We also analyze how the knowledge of query network properties can help decrease the cost of high availability.

Subfields: Computer Networks and Communications, Information Systems

View Paper: https://www.semanticscholar.org/paper/100a08f5997c72772a95c1c666e5247c6f2f376b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Information Systems

Citations in the corpus

The Design of the Borealis Stream Processing Engine (2005, background): “Replayability is useful in other contexts such as recovery and high availability [28].”
MillWheel: Fault-Tolerant Stream Processing at Internet Scale (2013, background): “Our use of sender-side buffering resembles “upstream backup” in [14], which also deﬁnes recovery semantics (precise, rollback, and gap) that mirror our own ﬂexible options for data delivery.”
Discretized streams: fault-tolerant streaming computation at scale (2013, methodology): “However, distributed streaming databases, such as Borealis, used replication or upstream backup for recovery [20].”
Discretized Streams: An Efficient and Fault-Tolerant Model for Stream Processing on Large Clusters (2012, background): “In upstream backup [12], each upstream node buffers the data sent to downstream nodes locally until it gets an acknowledgement that all related computations are done.”
Fault-tolerance in the Borealis distributed stream processing system (2005, background): “Previous schemes either do not address network failures [25] or strictly favor consistency over availability, by requiring at least one fully connected copy of the query network to exist to continue processing at any time [35].”