UbiCrawler: a scalable fully distributed Web crawler

By Paolo Boldi, Bruno Codenotti, Massimo Santini, Sebastiano Vigna. 2004

657 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Information Systems, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Signal Processing

Citations in the corpus

Perturbed Iterate Analysis for Asynchronous Stochastic Optimization (2015, methodology): “Following [6], we optimize a quadratic penalty relaxation for vertex cover eswiki-2013 where n ≈ 970K, and d ≈ 23M and wordassociation-2011 with n ≈ 10K and d ≈ 72K [18, 19, 20].”
Scalability! But at what COST? (2015, neutral): “name twitter rv [13] uk-2007-05 [5, 6] nodes 41,652,230 105,896,555 edges 1,468,365,182 3,738,733,648 size 5.”
The webgraph framework I: compression techniques (2004, methodology): “These data were either gathered from public sources (such as Web-Base [7]) or obtained with UbiCrawler [5, 4].”
Introduction to Information Retrieval (2008, positive): “Boldi et al. (2002) and Shkapenyuk and Suel (2002) provide more recent details of implementing large-scale distributed web crawlers.”
Main-memory triangle computations for very large (sparse (power-law)) graphs (2008, methodology): “It contains all the web pages in the .uk domain discovered during a crawl conducted from the 11-th of july, 2005, at 00:51, to the 30-th at 10:56 using UbiCrawler [11].”
GPS: a graph processing system (2013, methodology): “These datasets were provided by “The Labaratory for Web Algorithmics” [23], using software packages WebGraph [7], LLP [6] and UbiCrawler [5].”
Mizan: a system for dynamic load balancing in large-scale graph processing (2013, methodology): “[20] LAW.”
A smart local moving algorithm for large-scale modularity-based community detection (2013, methodology): “The crawl was performed using UbiCrawler (Boldi, Codenotti, Santini, & Vigna, 2004), and the network is made available by the Laboratory for Web Algorithmics at http://law.di.unimi.it.”
PowerLyra: Differentiated Graph Computation and Partitioning on Skewed Graphs (2019, neutral): “) [38] 42M 1.47B UK-2005 ( UK ) [4] 40M 936M Wiki ( WK ) [28] 5.7M 130M LJournal ( LJ ) [18] 5.4M 79M GWeb ( GW ) [42] 0.9M”
Direction-optimizing Breadth-First Search (2012, background): “The real social networks are taken from a variety of web crawls [6, 7, 12, 16, 21, 23, 25, 26].”
MG4J at TREC 2005 (2005, methodology): “MG4J is Java indexing system that we developed in the last thr ee years to support searches over the crawls performed by UbiCrawler [3].”