Detecting near-duplicates for web crawling

By Gurmeet Singh Manku, Arvind Jain, Anish Das Sarma. 2007

733 citations recorded by the source corpus.

Subfields: Information Systems, Information Retrieval & Search, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

In Information Retrieval & Search

In Signal Processing

Citations in the corpus

Asymmetric LSH (ALSH) for Sublinear Time Maximum Inner Product Search (MIPS) (2014, background): “or LSH is only dependent on ρ and n. Thus, LSH based near neighbor search in a sense does not suffer from the curse of dimensionality. This makes LSH a widely popular technique in industrial practice [14, 25, 7]. 2.3 LSH for L2 distance [9] presented a novel LSH family for all Lp (p ∈ (0,2]) distanc”
The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset (2023, methodology): “To remove near duplicate documents in OSCAR (which is already exact-deduplicated) we initially used SimHash (Charikar, 2002; Manku et al., 2007), a hashing function that associates to two similar texts hashes with a low Hamming distance, with 6-grams and a Hamming distance threshold of 4.”
b-Bit Minwise Hashing for Estimating Three-Way Similarities (2010, methodology): “These techniques have been successfully applied to many tasks in machine learning, databases, data mining, and information retrieval [18, 36, 11, 22, 16, 39, 28, 41, 27, 5, 2, 37, 7, 24, 21].”
One Permutation Hashing (2012, methodology): “In addition to near-neighbor search, these techniques are frequently used in popular operations like near-duplicate detection (Broder, 1997; Manku et al., 2007; Henzinger, 2006), all-pair similarity (Bayardo et al.”
C4Corpus: Multilingual Web-size Corpus with Free License (2016, result): “We follow a similar approach to the one introduced in (Manku et al., 2007).”
TOPSIG: topology preserving document signatures (2011, methodology): “This approach has been shown to reduce a similar index used for near duplicate detection by up to 50% when used with 64-bit codes [13].”
Efficient Interaction-based Neural Ranking with Locality Sensitive Hashing (2019, methodology): “Our work is motivated by the rich history of research on approximation with hashing such as LSH [1–4, 7, 8, 13, 15, 37] and in various data-intensive applications [6, 10, 17, 18, 30].”
Fast locality-sensitive hashing (2011, background): “Likewise, popular operations such as near-duplicate detection [7, 28, 19], all-pairs similarity [6], similarity join/record-linkage [25], temporal correlation [10], are made easier.”
Hashing for Similarity Search: A Survey (2014, methodology): “Specifically, The Altavista search engine uses min-hash to detect near-duplicate web pages [11], [12], while Google uses sim-hash to fulfill the same goal [92].”
OpenWebMath: An Open Dataset of High-Quality Mathematical Web Text (2023, methodology): “We use the SimHash implementation from text-dedup (Mou Preprint et al., 2023) to deduplicate the dataset using a threshold of 0.7.”
Deduplication of Scholarly Documents using Locality Sensitive Hashing and Word Embeddings (2020, methodology): “For example, locality sensitive hashing method has been successfully used for deduplication of web corpus (Manku et al., 2007), technical documentations (Forman et al., 2005) and clinical notes (Shenoy et al., 2017).”
Feature selection for text classification: A review (2018, methodology): “…measures for text processing, for example, Euclidean distance, Jaccard coefficient [54], Pearson correlation [40], Cosine Similarity [6], Hamming distance [62], Dice coefficient [85], IT-Sim [3, 56], SMTP [58], Earth mover’s distance [29, 96], Kullback-Leibler divergence [50], and BM25 [74].”
Learning and Inference via Maximum Inner Product Search (2016, methodology): “, to efficiently detect duplicate documents at internet scale (Henzinger, 2006; Manku et al., 2007).”
Densifying One Permutation Hashing via Rotation for Fast Near Neighbor Search (2014, methodology): “In addition to near-neighbor search, these techniques are frequently used in popular operations like near-duplicate detection (Broder, 1997; Manku et al., 2007; Henzinger, 2006), all-pair similarity (Bayardo et al.”