Large-scale Incremental Processing Using Distributed Transactions and Notifications

By Daniel Peng, Frank Dabek. USENIX Symposium on Operating Systems Design and Implementation, 2010

537 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Networks and Communications

View Paper: https://www.semanticscholar.org/paper/edc0e6a71118624d5ce887657301c6bf39df39df

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

Citations in the corpus

Naiad: a timely dataflow system (2013, background): “…three requirements: stream processors can produce low-latency re-sults for non-iterative algorithms [3, 5, 9, 38], batch systems can iterate synchronously at the expense of latency [27, 30, 43, 45], and trigger-based approaches support iteration with only weak consistency guarantees [29, 36, 46].”
MillWheel: Fault-Tolerant Stream Processing at Internet Scale (2013, background): “Google’s Percolator [27] also targets incremental updates to large datasets, but expects latencies on the order of minutes.”
Discretized Streams: An Efficient and Fault-Tolerant Model for Stream Processing on Large Clusters (2012, background): “Percolator [18] performs incremental computations using triggers, but does not offer consistency guarantees across nodes or high-level operators like map and join.”
Transactional storage for geo-replicated systems (2011, methodology): “Sinfonia uses transactions for infrastructure services [3, 4], while Percolator [38] uses them for search indexing.”
Efficient task replication for fast response times in parallel computation (2014, methodology): “…large scale computation in a data centers with a large number of computers, which is pioneered by companies like Google with the support from distributed computing frameworks such as MapReduce [9] and Percolator [15] , and distributed storage system such as Google File System [11] and BigTable [4].”
Exploiting iterative-ness for parallel ML computations (2014, background): “Percolator applies iterative refinement to a distributed key-value store with transactional updates and value triggers [34].”
Stale View Cleaning: Getting Fresh Answers from Stale Materialized Views (2015, background): “The increased research focus parallels a major concern in industrial systems for incrementally updating pre-computed results and indices such as Google Percolator [43] and Twitter’s Rainbird [49].”
Omega: flexible, scalable schedulers for large compute clusters (2013, methodology): “In this and subsequent graphs, solid lines represent batch jobs, and dashed lines are for service jobs. wide variety of jobs; some are con.gured by hand; some by automated systems such as MapReduce [8], Pregel [19] and Percolator [23].”
Spanner: Google's Globally-Distributed Database (2012, unknown): “Finally, the lack of cross-row transactions in Bigtable led to frequent complaints; Percolator [32] was in part built to address this failing.”
Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing (2012, methodology): “For these applications, it is more efficient to use systems that perform traditional update logging and data checkpointing, such as databases, RAMCloud [25], Percolator [26] and Piccolo [27].”
Kineograph: taking the pulse of a fast-changing and connected world (2012, methodology): “Google Percolator [24] provides a trigger-based eventdriven programming model for incremental web index construction.”
Occupy the cloud: distributed computing for the 99% (2017, result): “Why now? While the model described above is closely related to systems like Linda [6], Celias [15] and database trigger-based systems [35, 34], these systems have not been widely adopted.”
Discretized streams: fault-tolerant streaming computation at scale (2013, background): “Percolator [32] runs incremental computations using triggers, but does not offer high-level operators like map and join.”