NiagaraCQ: a scalable continuous query system for Internet databases

By Jianjun Chen, David J. DeWitt, Feng Tian, Yuan Wang. ACM SIGMOD Conference, 2000

1,305 citations recorded by the source corpus.

Continuous queries are persistent queries that allow users to receive new results when they become available. While continuous query systems can transform a passive web into an active environment, they need to be able to support millions of queries due to the scale of the Internet. No existing systems have achieved this level of scalability. NiagaraCQ addresses this problem by grouping continuous queries based on the observation that many web queries share similar structures. Grouped queries can share the common computation, tend to fit in memory and can reduce the I/O cost significantly. Furthermore, grouping on selection predicates can eliminate a large number of unnecessary query invocations. Our grouping technique is distinguished from previous group optimization approaches in the following ways. First, we use an incremental group optimization strategy with dynamic re-grouping. New queries are added to existing query groups, without having to regroup already installed queries. Second, we use a query-split scheme that requires minimal changes to a general-purpose query engine. Third, NiagaraCQ groups both change-based and timer-based queries in a uniform way. To insure that NiagaraCQ is scalable, we have also employed other techniques including incremental evaluation of continuous queries, use of both pull and push models for detecting heterogeneous data source changes, and memory caching. This paper presents the design of NiagaraCQ system and gives some experimental results on the system's performance and scalability.

Subfields: Artificial Intelligence, Computer Networks and Communications, Signal Processing

View Paper: https://www.semanticscholar.org/paper/23d6fce7d192aa54bd33845e1774dfdbb98f1ffa

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Signal Processing

Citations in the corpus

Continuous queries over data streams (2001, methodology): “Because continuous queries are longrunning, and some applications may involve a very large number of continuous queries, we expect that some form of multi-query optimization [Fin82, Sel88, CDTW00 ] will be a relevant and perhaps essential technique.”
Evaluating window joins over unbounded streams (2003, background): “One thrust in this body of research addresses problems arising when processing continuous queries [1][2].”
Delay aware querying with Seaweed (2006, neutral): “Many large-scale systems route tuples through longstanding pre-installed queries [2, 3, 10, 13, 23, 30].”
Fjording the stream: an architecture for queries over streaming sensor data (2002, background): “The NiagaraCQ project [5] is the most recent work focused on providing continuous queries over changing web sites.”
C-SPARQL: a Continuous Query Language for RDF Data Streams (2010, background): “OpenCQ [27] and NiagaraCQ [10) addressed continuous queries for monitoring persistent data sets spread over wid~area networks.”
Linear Road: A Stream Data Management Benchmark (2004, positive): “Stream data management has become a highly active research area and has inspired the development of several prototype systems including Aurora [1], STREAM [8], TelegraphCQ [4] and Niagara [5].”
Load shedding for aggregation queries over data streams (2004, background): “Some recent systems that incorporate stream processing include Aurora [5], Niagara [ 8 ], STREAM [21], and Telegraph [12], among many others.”
The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, Out-of-Order Data Processing (2015, background): “query systems [1, 14, 15], windowing [22], data streams [24], time domains [28], semantic models [9]), to the more recent forays in low-latency processing such as Spark Streaming [34], MillWheel, and Storm [5], modern consumers of data wield remarkable amounts of power in shaping and taming massive-”
Issues in data stream management (2003, background): “Limitations of traditional DBMSs in supporting streaming applications have been recognized, prompting research to augment existing technologies and build new systems to manage streaming data.”
Operator Scheduling in a Data Stream Manager (2003, background): “Neither NiagaraCQ nor STREAM has the notion of QoS.”
Flux: an adaptive partitioning operator for continuous query systems (2003, background): “Continuous query systems like [16, 7, 4] are essentially a collection of pipelined dataflows, a generalization of pipelined query plans, that are shared across numerous queries.”
Distributed top-k monitoring (2003, background): “…Chris Olston Stanford University {babcock, olston}@cs.stanford.edu ABSTRACT The querying and analysis of data streams has been a topic of much recent interest, motivated by applications from the .elds of networking, web usage analysis, sensor instrumentation, telecommunications, and others.”
The CQL continuous query language: semantic foundations and query execution (2006, methodology): “example, more sophisticated techniques such as [6, 20, 32] should be applicable.”