Finding near-duplicate web pages: a large-scale evaluation of algorithms

By Monika Henzinger. 2006

540 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Systems, Information Retrieval & Search, Algorithms & Complexity

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Information Systems

In Information Retrieval & Search

In Algorithms & Complexity

Citations in the corpus

Auto-grouping emails for faster e-discovery (2011, methodology): “It cannot be proved that these functions are min-wise independent [4] but they are found to be effective in practice [10] [13].”
Learning to Hash for Indexing Big Data—A Survey (2015, methodology): “, the min-hash approach outperforms other competing methods for the application of webpage duplicate detection [49].”
Self-taught hashing for fast similarity search (2010, background): “It is of great importance to many Information Retrieval (IR) [30] applications, such as near-duplicate detection [18], plagiarism analysis [43], collaborative filtering [26], caching [32], and content-based multimedia retrieval [28].”
Efficient Interaction-based Neural Ranking with Locality Sensitive Hashing (2019, methodology): “Our work is motivated by the rich history of research on approximation with hashing such as LSH [1–4, 7, 8, 13, 15, 37] and in various data-intensive applications [6, 10, 17, 18, 30].”
Effective and Fast Near Duplicate Detection via Signature-Based Compression Metrics (2016, methodology): “In [22], the author combined two algorithms, namely, shingling [2] and Charikar’s simhash [17], and achieved a better precision than each of the individuals.”
Asymmetric Minwise Hashing for Indexing Binary Inner Products and Set Containment (2015, background): “Later, because of its locality sensitive property [22], minhash became a widely used hash function for creating hash buckets leading to efficient algorithms for numerous applications including spam detection [6], collaborative filtering [4], news personalization [15], compressing social networks [13”
Mining of Massive Datasets (2011, methodology): “Two worth mentioning are [1] dealing with document analysis and [8] dealing with applications in Biology.”
Improved Densification of One Permutation Hashing (2014, background): “…minwise hashing is a popular indexing technique for a variety of large-scale data processing applications, which include duplicate detection [4, 13], all-pair similarity [3], fast linear learning [19], temporal correlation [10], 3-way similarity & retrieval [17, 23], graph algorithms [6, 11,…”
Large Scale Parallel Document Mining for Machine Translation (2010, methodology): “We train the systems on the Europarl Corpus (Koehn, 2002), the DGT Multilingual Translation Memory (European Commission Directorate-General for Translation, 2007) and the United Nations ODS corpus (United Nations, 2006).”
Efficient similarity joins for near duplicate detection (2008, background): “As a concrete example, a sizeable percentage of the Web pages are found to be near-duplicates by several studies [6, 14, 18].”
Data Matching (2012, background): “Because Web documents are commonly much larger than the records used in data matching and contain more detailed information, a main challenge is to extract relevant parts of documents to allow efficient indexing for scalable duplicate detection [139].”
Fast locality-sensitive hashing (2011, background): “Likewise, popular operations such as near-duplicate detection [7, 28, 19], all-pairs similarity [6], similarity join/record-linkage [25], temporal correlation [10], are made easier.”
Learning and Inference via Maximum Inner Product Search (2016, methodology): “This problem has numerous applications, including webpage duplication detection and image querying, where V is a set of vectorized web pages or images and q is a search query or webpage under question (Manku et al., 2007; Henzinger, 2006; Kulis & Grauman, 2009).”
Min-Max Kernels (2015, methodology): “In contrast, the resemblance kernel (2) has been widely used in practice on binary (or binarized) data [4, 5, 28, 9, 26, 7, 6, 11, 8, 16, 13, 1].”