Design and implementation of a high-performance distributed Web crawler
By Vladislav Shkapenyuk, Torsten Suel. Proceedings / International Conference on Data Engineering, 2002
418 citations recorded by the source corpus.
Subfields: Computer Networks and Communications, Information Systems
View Paper: https://www.semanticscholar.org/paper/e28e62ad1e345abf68ae96741d59433286916656
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Networks and Communications
- Paolo Boldi
- Sebastiano Vigna
- Massimo Santini
- Hector Garcia-Molina
- Junghoo Cho
- Bruno Codenotti
- Unnikrishna Pillai
- Seunghun Cha
In Information Systems
- Paolo Boldi
- Sebastiano Vigna
- Massimo Santini
- Junghoo Cho
- Hector Garcia-Molina
- Bruno Codenotti
- Torsten Suel
- Unnikrishna Pillai
Citations in the corpus
- Effective page refresh policies for Web crawlers (2003, background): “…focus on different issues, such as the crawler architecture [Heydon and Najork 1999; Page and Brin 1998], page selection [Menczer et al. 2001; Cho et al. 1998; Chakrabarti et al. 1999; Diligenti et al. 2000] crawler parallelization [Shkapenyuk and Suel 2002; Cho and Garcia-Molina 2002], etc.”
- UbiCrawler: a scalable fully distributed Web crawler (2004, methodology): “Even if no code is available, in several cases the basic design has been made public: this is the case, for instance, of Mercator [18] (the Altavista crawler), of the original Google crawler [6], and of some crawlers developed within the academic community [23, 24, 22].”
- The Perron-Frobenius theorem: some of its applications (2005, positive): “There are many system design and administration issues in such large scale runs that are outside the scope of this article; see [23] for details.”
- Introduction to Information Retrieval (2008, neutral): “Sindhwani and Keerthi (2006) present a more efficient implementation of a transductive SVM for large data sets. Tong and Koller (2001) explore active learning with SVMs for text classification; Baldridge and Osborne (2004) point out that examples selected for annotation with one classifier in an act”
- Web application security assessment by fault injection and behavior monitoring (2003, methodology): “We adopted many of the ideas and strategies reviewed in [19] and [58] to construct fast, parallel crawlers.”