Parallel crawlers

375 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Big Data: Survey, Technologies, Opportunities, and Challenges (2014, background): “In search engines, web crawler is a component that downloads and stores web pages [72].”
Toward Scalable Systems for Big Data Analytics: A Technology Tutorial (2014, background): “A crawler [72] is a program that downloads and stores webpages for a search engine.”
Exploiting the Block Structure of the Web for Computing (2003, background): “4Generally this requires a site-based crawler (such as the We bBase crawler [4]) which maintains a pool of active hosts, and craw ls hosts to completion before adding new hosts to the pool.”
UbiCrawler: a scalable fully distributed Web crawler (2004, methodology): “In the classification of [9], agents can use a different amount of coordination: at one extreme, all agents crawl the network independently, and one hopes that their overlap will be small due to a careful choice of the starting URLs; at the other extreme, a central coordinator divides the network ei”
Estimating and sampling graphs with multidimensional random walks (2010, methodology): “They are used to collect Web data [10], search P2P networks [6, 38], and decrease the time to discover “new wireless nodes” [2].”