The Evolution of the Web and Implications for an Incremental Crawler

By Junghoo Cho, Hector Garcia-Molina. Very Large Data Bases Conference, 2000

663 citations recorded by the source corpus.

View Paper: https://www.semanticscholar.org/paper/ed7087560de484b922e316874a9076376a2a0186

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Intelligent crawling on the World Wide Web with arbitrary predicates (2001, background): “Since the world wide web is rapidly evolving over time [12], it may often be desirable to repeat the same queries at di erent time periods.”
A large‐scale study of the evolution of Web pages (2003, methodology): “We observe signiﬁcant differences between top-level domains, conﬁrming earlier observations by Cho and Garcia-Molina [5].”
What's new on the web?: the evolution of the web from a search engine perspective (2004, methodology): “A number of existing studies have already investigated the evolution of the Web [16, 8, 18, 12, 15].”
Spam, damn spam, and statistics: using statistical analysis to locate spam web pages (2004, background): “Cho and Garcia-Molina [6] studied the fraction of pages on 270 web servers that changed day over day.”
Efficient Peer-to-Peer Keyword Searching (2003, background): “17 million unique words in105; 593 documents, retrieved by crawling to a recursion depth of five from 100 seed URLs [4].”
Topical web crawlers: Evaluating adaptive algorithms (2004, unknown): “General Terms: Algorithms, Design, Measurement, Performance Additional Key Words and Phrases: Ef.ciency, evaluation, evolution, exploitation, exploration, reinforcement learning, topical crawlers 1.”
Crawling the Hidden Web (2001, background): “These studies have addressed various issues, such as performance, scalability, freshness, extensibility, and parallelism, in the design and implementation of crawlers [3, 4, 6, 10, 17].”
Searching the Web (2001, background): “How should the crawling process be parallelized? Due to the enormous size of the Web, crawlers often run on multiple machines and download pages in parallel [10, 18].”
Estimating frequency of change (2003, methodology): “Reference [Cho and Garcia-Molina 2000a] proposes an architecture in which a crawler can adjust page revisit frequencies based on estimated page change frequencies.”